User Tag List

Thanks Thanks:  0
Dislikes Dislikes:  0
Page 2 of 3 FirstFirst 123 LastLast
Results 11 to 20 of 26

Thread: Working on RSS feeds and fetching content from other websites.

  1. #11
    Forum Admin
    Points: 7,342,646, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Overall activity: 99.8%
    Achievements:
    Three FriendsVeteranCreated Album picturesCreated Blog entryOverdriveTagger First ClassYour first GroupRecommendation First Class

    Working on RSS feeds and fetching content from other websites.


    TransAmDan's Avatar
    Join Date
    Apr 1999
    Location
    Waterlooville, Hampshire
    Posts
    13,126
    Points
    7,342,646
    Level
    100
    Total Contributions For

    TransAmDan     £ 5.00
    Avg. Time Online
    2 Hours 32 Minutes 32 Seconds
    Blog Entries
    31
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    vBActivity - Stats
    Points
    108,138
    Level
    75
    vBActivity - Bars
    Lv. Percent
    71.29%
    Disabled a day or so ago. It's fetching data, but not storing, perhaps database permissions or file access due to moving servers, will look at this and re-enable. There was some interesting articles and cars for sale popping up in there.

    2000 Trans-Am WS6 LS1 M6. (The Bat)
    Corsa Cat back, 3.5" dual tips. F.A.S.T LSX Intake / F.A.S.T Fuel rails, Dynatech LT Headers / Y-Pipe / No Cats, Wisper Lid, ported MAF, Racetronix 42lb/hr fuel injectors. Hypertech 160 stat. !AIR, !EGR. Walbro 255, SPEC Stage II Clutch / Flywheel. HPTuners 1BAR SD Tune. Wide Band gauge.
    389RWHP/380RWTQ ([477HP/460TQ]Flywheel) Dyno Dynamics

    More info and photos at:-
    http://www.solent-renegades.co.uk/co...-trans-am.html

  2. #12
    Forum Admin
    Points: 7,342,646, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Overall activity: 99.8%
    Achievements:
    Three FriendsVeteranCreated Album picturesCreated Blog entryOverdriveTagger First ClassYour first GroupRecommendation First Class

    Working on RSS feeds and fetching content from other websites.


    TransAmDan's Avatar
    Join Date
    Apr 1999
    Location
    Waterlooville, Hampshire
    Posts
    13,126
    Points
    7,342,646
    Level
    100
    Total Contributions For

    TransAmDan     £ 5.00
    Avg. Time Online
    2 Hours 32 Minutes 32 Seconds
    Blog Entries
    31
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    vBActivity - Stats
    Points
    108,138
    Level
    75
    vBActivity - Bars
    Lv. Percent
    71.29%
    Been looking at this all evening. Debugging many areas. It's getting stored as files and in the database correctly. However the generation of the XML file is adding <head/> near the top for the fun of it just to bugger things up.

    2000 Trans-Am WS6 LS1 M6. (The Bat)
    Corsa Cat back, 3.5" dual tips. F.A.S.T LSX Intake / F.A.S.T Fuel rails, Dynatech LT Headers / Y-Pipe / No Cats, Wisper Lid, ported MAF, Racetronix 42lb/hr fuel injectors. Hypertech 160 stat. !AIR, !EGR. Walbro 255, SPEC Stage II Clutch / Flywheel. HPTuners 1BAR SD Tune. Wide Band gauge.
    389RWHP/380RWTQ ([477HP/460TQ]Flywheel) Dyno Dynamics

    More info and photos at:-
    http://www.solent-renegades.co.uk/co...-trans-am.html

  3. #13
    Forum Admin
    Points: 7,342,646, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Overall activity: 99.8%
    Achievements:
    Three FriendsVeteranCreated Album picturesCreated Blog entryOverdriveTagger First ClassYour first GroupRecommendation First Class

    Working on RSS feeds and fetching content from other websites.


    TransAmDan's Avatar
    Join Date
    Apr 1999
    Location
    Waterlooville, Hampshire
    Posts
    13,126
    Points
    7,342,646
    Level
    100
    Total Contributions For

    TransAmDan     £ 5.00
    Avg. Time Online
    2 Hours 32 Minutes 32 Seconds
    Blog Entries
    31
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    vBActivity - Stats
    Points
    108,138
    Level
    75
    vBActivity - Bars
    Lv. Percent
    71.29%
    Been dabbling in this again. The glitch is caused on the new server, which is stopping the feeds coming into a thread.

    So the list of processes are
    • Fetch data from URL to extract links of articles, add link to database is doesn't already exist
    • Fetch page data from the link we just extracted, save page locally
    • Examine page and extract info, like title, contents, and URL for image
    • Fetch image and store local


    All of the above is working perfectly.

    Then when a datafeed is requested from the automatic RSS poster, it want to read an XML file, so I have written an XML export, so what a page is called it displays in XML. This worked fine before

    However at the top of the page I now get

    <?xml version="1.0" encoding="UTF-8"?>
    <head/><rss>
    <channel>
    <item>
    <articleid>1743</articleid>
    <feedid>2</feedid>

    The <head/> should not be there, it is confusing the XML reader(part of vbulletin) so how did it get there? Well its not in the source code. I have even written it as plain text, but when it gets passed though the system to be displayed as a page, it somehow gets added. There are slight variations of PHP versions between my testing server and the webserver.

    I have an idea, and it will certainly make things more efficient too.

    The articles are fetched and stored in the database to be exported in XML. However as they are in the database, and my code has access to the vbulletin database tables. Therefore I can write code to generate a thread and post the content. This will work out more efficient too. Just have to get the code right, with the speed of the server, i could create thousands of threads per second. I wont put any delete facility in my code. All will be fine.

    2000 Trans-Am WS6 LS1 M6. (The Bat)
    Corsa Cat back, 3.5" dual tips. F.A.S.T LSX Intake / F.A.S.T Fuel rails, Dynatech LT Headers / Y-Pipe / No Cats, Wisper Lid, ported MAF, Racetronix 42lb/hr fuel injectors. Hypertech 160 stat. !AIR, !EGR. Walbro 255, SPEC Stage II Clutch / Flywheel. HPTuners 1BAR SD Tune. Wide Band gauge.
    389RWHP/380RWTQ ([477HP/460TQ]Flywheel) Dyno Dynamics

    More info and photos at:-
    http://www.solent-renegades.co.uk/co...-trans-am.html

  4. #14
    Forum Admin
    Points: 7,342,646, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Overall activity: 99.8%
    Achievements:
    Three FriendsVeteranCreated Album picturesCreated Blog entryOverdriveTagger First ClassYour first GroupRecommendation First Class

    Working on RSS feeds and fetching content from other websites.


    TransAmDan's Avatar
    Join Date
    Apr 1999
    Location
    Waterlooville, Hampshire
    Posts
    13,126
    Points
    7,342,646
    Level
    100
    Total Contributions For

    TransAmDan     £ 5.00
    Avg. Time Online
    2 Hours 32 Minutes 32 Seconds
    Blog Entries
    31
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    vBActivity - Stats
    Points
    108,138
    Level
    75
    vBActivity - Bars
    Lv. Percent
    71.29%
    Made good progress on this last night and today. Automatically creates a thread and post entry. Updates counters. Limited it to 10 posts per hour as there are around 200 in the system already fetched.
    One thing I have noticed is they don't appear in 'What's New' button at the top. This must be another database table I need to update.

    2000 Trans-Am WS6 LS1 M6. (The Bat)
    Corsa Cat back, 3.5" dual tips. F.A.S.T LSX Intake / F.A.S.T Fuel rails, Dynatech LT Headers / Y-Pipe / No Cats, Wisper Lid, ported MAF, Racetronix 42lb/hr fuel injectors. Hypertech 160 stat. !AIR, !EGR. Walbro 255, SPEC Stage II Clutch / Flywheel. HPTuners 1BAR SD Tune. Wide Band gauge.
    389RWHP/380RWTQ ([477HP/460TQ]Flywheel) Dyno Dynamics

    More info and photos at:-
    http://www.solent-renegades.co.uk/co...-trans-am.html

  5. #15
    Premier Users
    Points: 61,348, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Overall activity: 43.0%
    Achievements:
    Tagger First ClassCreated Album picturesThree FriendsVeteranRecommendation Second Class50000 Experience Points
    Awards:
    Master Tagger

    Working on RSS feeds and fetching content from other websites.


    an88fiero's Avatar
    Join Date
    Aug 2013
    Location
    Swindon
    Posts
    3,675
    Points
    61,348
    Level
    100
    Total Contributions For

    an88fiero     £ 15.00
    Avg. Time Online
    44 Minutes 35 Seconds
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    vBActivity - Stats
    Points
    8,400
    Level
    27
    vBActivity - Bars
    Lv. Percent
    69.21%
    Quote Originally Posted by TransAmDan View Post
    Made good progress on this last night and today. Automatically creates a thread and post entry. Updates counters. Limited it to 10 posts per hour as there are around 200 in the system already fetched.
    One thing I have noticed is they don't appear in 'What's New' button at the top. This must be another database table I need to update.
    Dan, do you mean the "new posts" link up top? If so it does show them.
    Clare
    Swindon
    1997 Chevrolet Astro (my toy)
    2009 Dodge Challenger SRT8 (hubby's toy!)

  6. #16
    Forum Admin
    Points: 7,342,646, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Overall activity: 99.8%
    Achievements:
    Three FriendsVeteranCreated Album picturesCreated Blog entryOverdriveTagger First ClassYour first GroupRecommendation First Class

    Working on RSS feeds and fetching content from other websites.


    TransAmDan's Avatar
    Join Date
    Apr 1999
    Location
    Waterlooville, Hampshire
    Posts
    13,126
    Points
    7,342,646
    Level
    100
    Total Contributions For

    TransAmDan     £ 5.00
    Avg. Time Online
    2 Hours 32 Minutes 32 Seconds
    Blog Entries
    31
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    vBActivity - Stats
    Points
    108,138
    Level
    75
    vBActivity - Bars
    Lv. Percent
    71.29%
    Ah yes, so it does. That's good then.

    There is another button on the main menu 'Whats New' I think that one is called activity stream. It used to also show all new posts in there too.

    2000 Trans-Am WS6 LS1 M6. (The Bat)
    Corsa Cat back, 3.5" dual tips. F.A.S.T LSX Intake / F.A.S.T Fuel rails, Dynatech LT Headers / Y-Pipe / No Cats, Wisper Lid, ported MAF, Racetronix 42lb/hr fuel injectors. Hypertech 160 stat. !AIR, !EGR. Walbro 255, SPEC Stage II Clutch / Flywheel. HPTuners 1BAR SD Tune. Wide Band gauge.
    389RWHP/380RWTQ ([477HP/460TQ]Flywheel) Dyno Dynamics

    More info and photos at:-
    http://www.solent-renegades.co.uk/co...-trans-am.html

  7. #17
    Forum Admin
    Points: 7,342,646, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Overall activity: 99.8%
    Achievements:
    Three FriendsVeteranCreated Album picturesCreated Blog entryOverdriveTagger First ClassYour first GroupRecommendation First Class

    Working on RSS feeds and fetching content from other websites.


    TransAmDan's Avatar
    Join Date
    Apr 1999
    Location
    Waterlooville, Hampshire
    Posts
    13,126
    Points
    7,342,646
    Level
    100
    Total Contributions For

    TransAmDan     £ 5.00
    Avg. Time Online
    2 Hours 32 Minutes 32 Seconds
    Blog Entries
    31
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    vBActivity - Stats
    Points
    108,138
    Level
    75
    vBActivity - Bars
    Lv. Percent
    71.29%
    I found the table that controls that bit. Called Activity stream. I have most of the info for it apart from ContentID, I have other ID, but they not relevant. I won't drive into this further at this time. Could be opening a can of wormies.

    2000 Trans-Am WS6 LS1 M6. (The Bat)
    Corsa Cat back, 3.5" dual tips. F.A.S.T LSX Intake / F.A.S.T Fuel rails, Dynatech LT Headers / Y-Pipe / No Cats, Wisper Lid, ported MAF, Racetronix 42lb/hr fuel injectors. Hypertech 160 stat. !AIR, !EGR. Walbro 255, SPEC Stage II Clutch / Flywheel. HPTuners 1BAR SD Tune. Wide Band gauge.
    389RWHP/380RWTQ ([477HP/460TQ]Flywheel) Dyno Dynamics

    More info and photos at:-
    http://www.solent-renegades.co.uk/co...-trans-am.html

  8. #18
    Premier Users
    Points: 61,348, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Overall activity: 43.0%
    Achievements:
    Tagger First ClassCreated Album picturesThree FriendsVeteranRecommendation Second Class50000 Experience Points
    Awards:
    Master Tagger

    Working on RSS feeds and fetching content from other websites.


    an88fiero's Avatar
    Join Date
    Aug 2013
    Location
    Swindon
    Posts
    3,675
    Points
    61,348
    Level
    100
    Total Contributions For

    an88fiero     £ 15.00
    Avg. Time Online
    44 Minutes 35 Seconds
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    vBActivity - Stats
    Points
    8,400
    Level
    27
    vBActivity - Bars
    Lv. Percent
    69.21%
    Are you leaving the RSS feeds to show in the "new posts" link?
    Clare
    Swindon
    1997 Chevrolet Astro (my toy)
    2009 Dodge Challenger SRT8 (hubby's toy!)

  9. #19
    Forum Admin
    Points: 7,342,646, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Overall activity: 99.8%
    Achievements:
    Three FriendsVeteranCreated Album picturesCreated Blog entryOverdriveTagger First ClassYour first GroupRecommendation First Class

    Working on RSS feeds and fetching content from other websites.


    TransAmDan's Avatar
    Join Date
    Apr 1999
    Location
    Waterlooville, Hampshire
    Posts
    13,126
    Points
    7,342,646
    Level
    100
    Total Contributions For

    TransAmDan     £ 5.00
    Avg. Time Online
    2 Hours 32 Minutes 32 Seconds
    Blog Entries
    31
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    vBActivity - Stats
    Points
    108,138
    Level
    75
    vBActivity - Bars
    Lv. Percent
    71.29%
    Quote Originally Posted by an88fiero View Post
    Are you leaving the RSS feeds to show in the "new posts" link?
    Yes, was thinking of leaving it how it is at the mo.
    Unless you think it's not good to do so. At the moment the feeds are playing catch up. I think it will be only 3 per hour once it's settled down.

    2000 Trans-Am WS6 LS1 M6. (The Bat)
    Corsa Cat back, 3.5" dual tips. F.A.S.T LSX Intake / F.A.S.T Fuel rails, Dynatech LT Headers / Y-Pipe / No Cats, Wisper Lid, ported MAF, Racetronix 42lb/hr fuel injectors. Hypertech 160 stat. !AIR, !EGR. Walbro 255, SPEC Stage II Clutch / Flywheel. HPTuners 1BAR SD Tune. Wide Band gauge.
    389RWHP/380RWTQ ([477HP/460TQ]Flywheel) Dyno Dynamics

    More info and photos at:-
    http://www.solent-renegades.co.uk/co...-trans-am.html

  10. #20
    Premier Users
    Points: 61,348, Level: 100
    Level completed: 0%, Points required for next Level: 0
    Overall activity: 43.0%
    Achievements:
    Tagger First ClassCreated Album picturesThree FriendsVeteranRecommendation Second Class50000 Experience Points
    Awards:
    Master Tagger

    Working on RSS feeds and fetching content from other websites.


    an88fiero's Avatar
    Join Date
    Aug 2013
    Location
    Swindon
    Posts
    3,675
    Points
    61,348
    Level
    100
    Total Contributions For

    an88fiero     £ 15.00
    Avg. Time Online
    44 Minutes 35 Seconds
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    vBActivity - Stats
    Points
    8,400
    Level
    27
    vBActivity - Bars
    Lv. Percent
    69.21%
    Leave it as it is Dan. If it gets too much I can find another way to view new posts

    Thanks
    Clare
    Swindon
    1997 Chevrolet Astro (my toy)
    2009 Dodge Challenger SRT8 (hubby's toy!)

Similar Threads

  1. How other websites/forums perform...
    By TransAmDan in forum Comments
    Replies: 2
    Last Post: 08-08-2015, 08:35
  2. Google Now Favours Websites Using SSL
    By TransAmDan in forum Recent Changes and Adjustments
    Replies: 5
    Last Post: 10-02-2015, 22:25
  3. Working on RSS Feeds
    By TransAmDan in forum Recent Changes and Adjustments
    Replies: 1
    Last Post: 14-12-2014, 17:00
  4. Better RSS Feeds
    By TransAmDan in forum Recent Changes and Adjustments
    Replies: 1
    Last Post: 03-01-2014, 10:57
  5. Site Speed up, going to try Content Delivery Network.
    By TransAmDan in forum Recent Changes and Adjustments
    Replies: 4
    Last Post: 20-04-2012, 12:36

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •