Historical data stream provides fragmented data

[mcg05Participant]

Hi everyone,

Last week, I signed up to the TFL website to access the historical “Barclays Cycle Hire statistics feed” and I immediately received an email with a link to the feed.

The link allowed me to download a 10.6mb zip file (~40mb of CVS). However, instead of the advertised

“31 May 2011 to 4 February 2012″ data the zip file contained 4 files worth of data for the following dates: 01/02/12-06/02/12, 29/02/12-02/03/12, 01/04/12-15/04/12 and 11/05/12-24/05/12. Each file only contained 65536 entries, which seems rather low given the weekly stats shown on (http://www.tfl.gov.uk/roadusers/cycling/20389.aspx)

Has anyone else experienced this? Also does anyone know where/how to get the actual “31 May 2011 to 4 February 2012″ data containing all journeys?

Cheers

Chris

[BimblesParticipant]

65536? Isn’t that the old Excel row limit?

Could say something about how they extracted the data…

[mcg05Participant]

That could be the case, indeed. To be sure it wasn’t my OpenOffice causing the issue I checked the files in VIM and again I only got 65536 entries in each of the 4 files.

I have contacted TFL in this matter.

In the mean time – would anyone mind sharing their “31 May 2011 to 4 February 2012″ data set with me?

[MattParticipant]

I don’t think I have a copy I’m afraid. If you need more recent data (the last few weeks) let me know and I’ll export it from my database.

It has got to 2.5m rows now though…!

[mcg05Participant]

@blighter. Thank you for the kind offer. I was just about to enquire as to how we could best exchange the data when I got the following email from TfL:

Code:

Hi Chris

Thanks for your email.

The increasing popularity of the scheme and the size of the data means we can not back date as far as we used to, we will go back as far as we can an make sure we have a complete dataset and tat the date ranges are correctly specified.

This work will take up to 2 weeks to complete.

Thanks

TfL Online

From: Guenther, Chris
To: Developers
Subject: Missing cycle hire data

Hi,

I registered for the following data stream:

“Barclays Cycle Hire statistics” – Details of all Barclays Cycle Hire journeys made from 31 May 2011 to 4 February 2012

and I subsequently received an email with the following link:

However after I download the file I realised that it only contains 4 CSV files with 65536 hire entries
each (dates: 01/02/12-06/02/12, 29/02/12-02/03/12, 01/04/12-15/04/12 and 11/05/12-24/05/12).

Is it still possible to get the data of all the journeys made “from 31 May 2011 to 4 February 2012″
or a similar more recent data set say “from 4 February 2012 until 1 June 2012″?

Kind regards
Chris

I will try it again in two weeks time, but I might come back to your offer if it doesn’t work.

Cheers

Chris

[jamesinealingParticipant]

Replying to @mcg05‘s post:

Can I just ask, I signed up for the stats but the latest data is 25/7 and the API page says n/a against update frequency. Does anyone know in practise how frequently they update it, and what the typical lag is in the data they provide?

Thanks

[jamesinealingParticipant]

This response from TFL:

“They update every 3-6 months, we are just waiting for the data to the end of October to be ready – it should be up by the end of the month. Please note we only release 6 months of data, so if you require historical data please ensure you save the current dataset, as Feb-Apr data will soon be removed.”

I’m seeking clarification exactly what they mean by this!

On the original point, the 6 files now look to have a complete date range, though 29/02-03/03 appear to be duplicate across parts 1 & 2, and there are some very odd records with dates of 1900 and 1901 in I think part 4!

[Author]

Posts