Error in loop

[mk1564]

Hi,

Thank you very much for sharing this nice code. I am trying to use loop to get ALFRED data using your function. At some point in the loop, I get error message like below. When I manually execute the command at that point, it does work. Can I ask why this happens? Is it because of I am extracting data from the server too fast?

Error using xmlread (line 116)
Java exception occurred:
java.io.IOException: Server returned HTTP response code: 429 for URL:
https://api.stlouisfed.org/fred/series?series_id=COMPNFB&observation_start=1949-10-01&observation_end=2015-12-02&units=lin&frequency=q&aggregation_method=avg&realtime_start=2015-12-02&realtime_end=2015-12-02&api_key=190e4c2ccb2c648cd2cf41d0b1999b96

	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1894)

	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)

	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:263)

	at org.apache.xerces.impl.XMLEntityManager.setupCurrentEntity(Unknown Source)

	at org.apache.xerces.impl.XMLVersionDetector.determineDocVersion(Unknown Source)

	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)

	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)

	at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)

	at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)

Error in getFredData (line 116)
xDoc=xmlread(['https://api.stlouisfed.org/fred/series?series_id=',series_id,optionstring,'&api_key=',api_key]); 

[mk1564]

Here is a piece of code for getting GDPC1. I am trying to create a table that matches up the exciting data format.

%% GDPC1
T_y = df_tfp(idxstart:end,1); % output

for i = 2:width(df_tfp)
    ondate = df_tfp.Properties.VariableNames{:,i};
    ondate = datetime(ondate,'Inputformat','dd-MMM-yyyy');
    ondate = datestr(ondate,'yyyy-mm-dd');
    
    df_y = getFredData('GDPC1',startdate,ondate,'log','q','avg',ondate);
    y = df_y.Data(:,2);
    y = [y;NaN(height(T_y)-length(y),1)];
    Varname_y = df_tfp.Properties.VariableNames{:,i};
    T_y.(Varname_y) = y;
end

[robertdkirkby]

I don't really understand the error but I believe you are correct when you say "Is it because of I am extracting data from the server too fast?"

In my experience this error occurs when you request lots of series in a loop. I do not understand the cause but believe it is something at the 'FRED end'. Typically if you just wait a half a minute and then run the command again it will then work. I sometimes do this manually by just having the loop print the counter (i in your example) and then rerun from there until the end. The other alternative that might work would be to put a pause() inside the loop. Obviously this would make the code slower, but getting data is probably not something you are having to run too often so you could do that and then just save the downloaded data to a mat-file for future use.

Sorry I can't be of more help but as I said, I don't really understand the cause of the error so all I have a workarounds.

If anyone reading this understands the actual cause please let me know.

[mk1564]

Thank you very much for your reply. I guess like you said it is because of "too many requests" 429 error. Thanks for suggestion for pause(n)! I am going to have to play around with n to avoid the error. Thank you very much again!