Convert Data with mixed datatypes to LibSVM format

[sumitsidana]

I have data with about Million rows and 3 columns. The columns are of 3 different datatypes. NumberOfFollowers is of a numerical datatype, UserName is of a categorical data type, Embeddings is of categorical-set type.

df:

Index  NumberOfFollowers                  UserName                    Embeddings        Target Variable

0        15                                name1                      [0.5 0.3 0.2]       0
1        4                                 name2                      [0.4 0.2 0.4]       1
2        8                                 name3                      [0.5 0.5 0.0]       0
3        10                                name1                      [0.1 0.0 0.9]       0
...      ...                               ....                       ...                 ..

I would like to convert this data into the LibSVM input format.

Desired Output:

0 0:15 4:1 1:0.5 2:0.3 3:0.2
1 0:4 5:1 1:0.4 2:0.2 3:0.4
0 0:8 6:1 1:0.5 2:0.5 3:0.0
0 0:10 4:1 1:0.1 2:0.0 3:0.9
...

The Perl script https://github.com/srendle/libfm/blob/master/scripts/triple_format_to_libfm.pl handles categorical values. But, how to handle mixture of data types as also described in this paper: https://www.ismll.uni-hildesheim.de/pub/pdfs/Rendle_et_al2011-Context_Aware.pdf

Can this problem be solved using libfm or I have to use external tools? If I need to use external tools, are you aware of any external tools which perform this operation on a very large scale data (as I have many columns of mixed data types)?

[ljluestc]

import pandas as pd
from sklearn.preprocessing import LabelEncoder
from sklearn.datasets import dump_svmlight_file

# Assuming 'df' is your DataFrame with the data
df = pd.DataFrame({
    'NumberOfFollowers': [15, 4, 8, 10],
    'UserName': ['name1', 'name2', 'name3', 'name1'],
    'Embeddings': [[0.5, 0.3, 0.2], [0.4, 0.2, 0.4], [0.5, 0.5, 0.0], [0.1, 0.0, 0.9]],
    'Target Variable': [0, 1, 0, 0]
})

# Preprocess 'UserName' - convert categorical data to numeric
le = LabelEncoder()
df['UserName'] = le.fit_transform(df['UserName'])

# Handle 'Embeddings' - each element in the list becomes a separate feature
# Assuming Embeddings are always 3 elements long for simplicity
embeddings_df = pd.DataFrame(df['Embeddings'].tolist(), columns=['Emb1', 'Emb2', 'Emb3'])
df = pd.concat([df.drop('Embeddings', axis=1), embeddings_df], axis=1)

# Prepare for LibSVM format
# Here, we assume 'NumberOfFollowers' is feature 0, 'UserName' is 4 (since we're using 1-based indexing), 
# and each embedding element is 1, 2, and 3 respectively.
X = df.drop('Target Variable', axis=1)
y = df['Target Variable']

# Convert to LibSVM format
# zero_based=False because LibSVM uses 1-based indexing
dump_svmlight_file(X, y, 'output.libsvm', zero_based=False)

# Print the first few lines to check the format
with open('output.libsvm', 'r') as file:
    for _ in range(4):  # Print first 4 lines
        print(file.readline().strip())