Public Data Access Overview / Discussion Forum

Hello,

I've seen this issue has been brought up before but I can't see a clear answer regarding the source or resolution.

I'm attempting to download specific quantities from the snapshot zero snapshot and group catalogue files. Sometimes this works, but sometimes I get "403 Client Error: Forbidden for url". This also occurs when attempting to download files that I have successfully downloaded using the same process before.

Does anyone know the source of this error and/or a way to resolve it? I've copied the code I'm using below if needed.

Thanks,
Jack

import os
import requests
import h5py
import numpy as np

def get(path, key, params=None, outdir='./'):

    # make HTTP GET request to path
    headers = {"api-key":key}
    r = requests.get(path, params=params, headers=headers)

    # raise exception if response code is not HTTP SUCCESS (200)
    r.raise_for_status()

    if r.headers['content-type'] == 'application/json':
        return r.json() # parse json responses automatically

    if 'content-disposition' in r.headers:
        name = r.headers['content-disposition'].split("filename=")[1]
        filename = f'{outdir}/{name}'
        with open(filename, 'wb') as f:
            f.write(r.content)
        return filename # return the filename string

    return r

key = "key_goes_here"

snapshot = 0 

outdir = './'

base_url = "http://www.tng-project.org/api/TNG100-1/"
sim_metadata = get(base_url, key)
n_files = sim_metadata['num_files_snapshot']

params = {'stars':'Coordinates,GFM_Metallicity,Masses,ParticleIDs,GFM_StellarFormationTime', 'gas':'Coordinates,GFM_Metallicity,Masses,ParticleIDs'}

for i in range(n_files):
    file_url = f'{base_url}/files/snapshot-{snapshot}.{i}.hdf5'
    saved_filename = get(file_url, key, params=params, outdir=outdir)

params_all = [{'Subhalo':'SubhaloGrNr'}, {'Subhalo':'SubhaloLenType'}, {'Group':'GroupFirstSub'}, {'Group':'GroupLenType'}, {'Group':'GroupNsubs'}]

filenames = []
for param in params_all:
    file_url = f'{base_url}/files/groupcat-{snapshot}/?{list(param.keys())[0]}={param[list(param.keys())[0]]}'
    filename = get(file_url, key, params=None, outdir=outdir)

    filenames.append(filename)

with h5py.File(f'{outdir}/groupcat-{snapshot}.hdf5', 'w') as cat:

    for filename, param in zip(filenames, params_all):
        with h5py.File(filename, 'r') as f:
            for key, value in param.items():
                cat.require_group(key)
                cat[f'{key}/{value}'] = f[f'{key}/{value}'][:]
        os.remove(filename)

If this issue only occurs sometimes, and is not reproducible, it is likely due to times of high load.

I would build into your script automatic error detection, and automatic retry.