We have recently released the Observed Antibody Space database – collection of cleaned and annotated antibody sequence (Ig-seq or AIRR-seq) data from 53 studies. We have formatted the data in the format that should facilitate data mining and since release we had several queries on how to parse the data out. Therefore here we give a small example of how to parse the data and make sense of it.
You should download the bulk data file from OAS, available here.
The datasets are separated into ‘data units‘ – collections of sequences that can be uniquely assigned to a range of metadata parameters such as study, organism etc. Our task therefore is to iterate through all those files and read sequences from each of these. Firstly we will attempt to iterate through the files and I will assume that you uncompressed the bulk data file into ../data/json folder. We will write a helper function that will simply list all files with its full paths in a directory and call it list_file_paths
import os
#Fetch all files in directory and subdirectories.
def list_file_paths(directory):
for dirpath,_,filenames in os.walk(directory):
for f in filenames:
yield os.path.abspath(os.path.join(dirpath, f))
if __name__ == '__main__':
#Replace this with the location of where you uncompressed the bulk data file.
directory = '../data/json'
for f in list_file_paths(directory):
print f
The code above will list all the files in ../data/json which incidentally are all the ‘data units’. Now our task is to parse out the output from each of the data units. They are gzipped files with data element on each line. Therefore we will use gzip library to stream the contents of a gzipped file rather than uncompressing each one of them separately. This is achieved by function parse_single_file
import os,gzip
#Fetch all files in directory and subdirectories.
def list_file_paths(directory):
for dirpath,_,filenames in os.walk(directory):
for f in filenames:
yield os.path.abspath(os.path.join(dirpath, f))
#Parse out the contents of a single file.
def parse_single_file(src):
#The first line are the meta entries.
meta_line = True
for line in gzip.open(src,'rb'):
print line
if __name__ == '__main__':
#Replace this with the location of where you uncompressed the bulk data file.
directory = '../data/json'
for f in list_file_paths(directory):
parse_single_file(f)
The code above will simply go through all the data unit files, stream the gzipped lines and print each one of them separately. Each line however is formatted as json – meaning it can be parsed using pythons json library and act as a pythonic dictionary. below we have parsed out the basic elements in the final incarnation of the code:
import os,gzip,json,pprint
#Fetch all files in directory and subdirectories.
def list_file_paths(directory):
for dirpath,_,filenames in os.walk(directory):
for f in filenames:
yield os.path.abspath(os.path.join(dirpath, f))
#Parse out the contents of a single file.
def parse_single_file(src):
#The first line are the meta entries.
meta_line = True
for line in gzip.open(src,'rb'):
if meta_line == True:
metadata = json.loads(line)
meta_line=False
print "Metadata:"
pprint.pprint(metadata)
continue
#Parse actual sequence data.
basic_data = json.loads(line)
print "Basic data:"
pprint.pprint(basic_data)
#IMGT-Numbered sequence.
print "IMGT-numbered sequence"
d = json.loads(basic_data['data'])
pprint.pprint(d)
print "==========="
if __name__ == '__main__':
#Replace this with the location of where you uncompressed the bulk data file.
directory = '../data/json'
for f in list_file_paths(directory):
parse_single_file(f)
The first line of each data unit are meta entries. These look as follows:
{u'Age': u'22-70',
u'Author': u'Halliley et al., (2015)',
u'BSource': u'Bone-Marrow',
u'BType': u'Plasma-B-Cells',
u'Chain': u'Heavy',
u'Disease': u'None',
u'Isotype': u'IGHM',
u'Link': u'https://doi.org/10.1016/j.immuni.2015.06.016',
u'Longitudinal': u'no',
u'Size': 934,
u'Species': u'human',
u'Subject': u'no',
u'Vaccine': u'Tetanus/Flu'}
The attributes should be self-explanatory and the existence of this data on top of each file is supposed to streamline searching through data-units if you wish to parse sequences given a particular configuration of meta-data entries (e.g. organism).
Next, the code parses out data on each sequence that is associated with its genes, full sequence, CDR3 and numbered sequence. Therefore the output for this will look something like this:
{u'cdr3': u'ARHQGVYWVTTAGLSH',
u'data': u'{"fwh1": {"11": "G", "24": "T", "13": "V", "12": "L", "15": "P", "14": "K", "17": "E", "16": "S", "19": "L", "18": "T", "22": "T", "26": "S", "25": "V", "21": "L", "20": "S", "23": "C"}, "fwh3": {"68": "N", "88": "S", "89": "L", "66": "Y", "67": "Y", "82": "T", "83": "S", "80": "V", "81": "D", "86": "Q", "87": "F", "84": "K", "85": "N", "92": "S", "79": "S", "69": "P", "104": "C", "78": "I", "77": "T", "76": "V", "75": "R", "74": "S", "72": "K", "71": "L", "70": "S", "102": "Y", "90": "K", "100": "A", "101": "V", "95": "T", "94": "V", "97": "A", "96": "A", "91": "L", "99": "T", "98": "D", "93": "S", "103": "Y"}, "fwh2": {"52": "W", "39": "W", "48": "Q", "49": "G", "46": "P", "47": "G", "44": "Q", "45": "P", "51": "E", "43": "R", "40": "G", "42": "I", "55": "S", "53": "I", "54": "G", "41": "W", "50": "L"}, "fwh4": {"120": "Q", "121": "G", "122": "T", "123": "L", "124": "V", "125": "P", "126": "V", "127": "S", "128": "S", "119": "G", "118": "W"}, "cdrh1": {"27": "G", "37": "Y", "31": "S", "30": "I", "28": "G", "29": "S", "35": "S", "34": "S", "38": "Y", "36": "S"}, "cdrh2": {"59": "S", "58": "Y", "57": "S", "56": "I", "63": "G", "64": "T", "65": "T"}, "cdrh3": {"111A": "W", "109": "G", "108": "Q", "115": "L", "114": "G", "117": "H", "116": "S", "111": "Y", "110": "V", "113": "A", "112": "T", "112A": "T", "112B": "V", "106": "R", "107": "H", "105": "A"}}',
u'j': u'IGHJ1*01',
u'name': 12,
u'redundancy': 1,
u'seq': u'GLVKPSETLSLTCTVSGGSISSSSYYWGWIRQPPGQGLEWIGSISYSGTTYYNPSLKSRVTISVDTSKNQFSLKLSSVTAADTAVYYCARHQGVYWVTTAGLSHWGQGTLVPVSS',
u'v': u'IGHV4-39*07'}
Above, redundancy refers to how many times we see a given sequence (seq) in a particular study. We also store the IMGT-numbered data (the data attribute) which needs a second round of json parsing and its output is a dictionary of IMGT-number – amino acid associations grouped by the regions of an antibody (cdrs and framework regions):
{u'cdrh1': {u'27': u'G',
u'28': u'G',
u'29': u'S',
u'30': u'I',
u'31': u'S',
u'34': u'S',
u'35': u'S',
u'36': u'S',
u'37': u'Y',
u'38': u'Y'},
u'cdrh2': {u'56': u'I',
u'57': u'S',
u'58': u'Y',
u'59': u'S',
u'63': u'G',
u'64': u'T',
u'65': u'T'},
u'cdrh3': {u'105': u'A',
u'106': u'R',
u'107': u'H',
u'108': u'Q',
u'109': u'G',
u'110': u'V',
u'111': u'Y',
u'111A': u'W',
u'112': u'T',
u'112A': u'T',
u'112B': u'V',
u'113': u'A',
u'114': u'G',
u'115': u'L',
u'116': u'S',
u'117': u'H'},
u'fwh1': {u'11': u'G',
u'12': u'L',
u'13': u'V',
u'14': u'K',
u'15': u'P',
u'16': u'S',
u'17': u'E',
u'18': u'T',
u'19': u'L',
u'20': u'S',
u'21': u'L',
u'22': u'T',
u'23': u'C',
u'24': u'T',
u'25': u'V',
u'26': u'S'},
u'fwh2': {u'39': u'W',
u'40': u'G',
u'41': u'W',
u'42': u'I',
u'43': u'R',
u'44': u'Q',
u'45': u'P',
u'46': u'P',
u'47': u'G',
u'48': u'Q',
u'49': u'G',
u'50': u'L',
u'51': u'E',
u'52': u'W',
u'53': u'I',
u'54': u'G',
u'55': u'S'},
u'fwh3': {u'100': u'A',
u'101': u'V',
u'102': u'Y',
u'103': u'Y',
u'104': u'C',
u'66': u'Y',
u'67': u'Y',
u'68': u'N',
u'69': u'P',
u'70': u'S',
u'71': u'L',
u'72': u'K',
u'74': u'S',
u'75': u'R',
u'76': u'V',
u'77': u'T',
u'78': u'I',
u'79': u'S',
u'80': u'V',
u'81': u'D',
u'82': u'T',
u'83': u'S',
u'84': u'K',
u'85': u'N',
u'86': u'Q',
u'87': u'F',
u'88': u'S',
u'89': u'L',
u'90': u'K',
u'91': u'L',
u'92': u'S',
u'93': u'S',
u'94': u'V',
u'95': u'T',
u'96': u'A',
u'97': u'A',
u'98': u'D',
u'99': u'T'},
u'fwh4': {u'118': u'W',
u'119': u'G',
u'120': u'Q',
u'121': u'G',
u'122': u'T',
u'123': u'L',
u'124': u'V',
u'125': u'P',
u'126': u'V',
u'127': u'S',
u'128': u'S'}}
We hope this quick intro to our data format will allow you to do great science with this data.
