This repository contains a PySpark-based recommendation system using Alternating Least Squares (ALS) for collaborative filtering. The goal is to predict user ratings for movies and provide recommendations based on those predictions. The dataset used in this project is the MovieLens dataset, which contains movie ratings from users.
Recommendation systems are widely used in various applications like e-commerce, streaming services, and social media platforms. Collaborative filtering is one of the most popular techniques used in recommendation systems. ALS (Alternating Least Squares) is a matrix factorization algorithm that is particularly well-suited for collaborative filtering tasks.
This project demonstrates how to implement an ALS-based recommendation system using PySpark. The code includes data loading, preprocessing, model training, evaluation, and making recommendations.
The dataset used in this project is the MovieLens dataset, which includes:
- ratings.csv: Contains user ratings for movies.
- movies.csv: Contains movie titles and genres.
-
ratings.csv:
userId: Unique identifier for the user.movieId: Unique identifier for the movie.rating: Rating given by the user to the movie.timestamp: Timestamp when the rating was given.
-
movies.csv:
movieId: Unique identifier for the movie.title: Title of the movie.genres: Genres associated with the movie.
The code starts by setting up a PySpark session. PySpark is the Python API for Apache Spark, which is a distributed computing framework. The session is configured with specific memory settings to handle large datasets efficiently.
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName('Recommender_system') \
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") \
.config("spark.kryoserializer.buffer.max", "1024m") \
.getOrCreate()The dataset is loaded into PySpark DataFrames. The ratings.csv and movies.csv files are read and their schemas are inferred.
ratings = spark.read.csv("/Users/ysk/Downloads/archive(3)/ratings.csv", header=True, inferSchema=True)
movies = spark.read.csv("/Users/ysk/Downloads/archive(3)/movies.csv", header=True, inferSchema=True)Basic data exploration is performed to understand the dataset. This includes checking the schema, counting the number of rows, and displaying the first few rows of the dataset.
ratings.printSchema()
movies.printSchema()
print("Ratings dataset shape: Rows =", ratings.count(), ", Columns =", len(ratings.columns))
print("Movies dataset shape: Rows =", movies.count(), ", Columns =", len(movies.columns))The dataset is split into training and testing sets. The training set is used to train the ALS model, and the testing set is used to evaluate the model's performance.
train_data, test_data = ratings.randomSplit([0.7, 0.3], seed=5033)The ALS model is trained using the training dataset. The model is configured with parameters such as the number of iterations, regularization parameter, and columns for user, item, and rating.
from pyspark.ml.recommendation import ALS
als = ALS(maxIter=5, regParam=0.01, userCol="userId", itemCol="movieId", ratingCol="rating")
model = als.fit(train_data)The model's performance is evaluated using the Root Mean Squared Error (RMSE) metric. The predictions are made on the test dataset, and the RMSE is calculated.
from pyspark.ml.evaluation import RegressionEvaluator
predictions = model.transform(test_data)
evaluator = RegressionEvaluator(metricName="rmse", labelCol="rating", predictionCol="prediction")
rmse = evaluator.evaluate(predictions)
print(f'Root Mean Squared Error (RMSE): {rmse}')The trained model is used to make recommendations for a specific user. The recommendations are sorted by the predicted rating.
user_recommendations = model.recommendForAllUsers(10)
user_recommendations.show()Alternating Least Squares (ALS) is a matrix factorization algorithm used in collaborative filtering. It decomposes the user-item interaction matrix into two lower-dimensional matrices: one representing users and the other representing items. The goal is to minimize the difference between the observed ratings and the predicted ratings.
-
Matrix Factorization: The user-item interaction matrix ( R ) is factorized into two matrices ( U ) (user factors) and ( V ) (item factors) such that ( R \approx U \times V^T ).
-
Alternating Optimization: The algorithm alternates between fixing ( U ) and optimizing ( V ), and fixing ( V ) and optimizing ( U ). This process continues until convergence.
-
Regularization: To prevent overfitting, a regularization term is added to the loss function.
- Scalability: ALS is highly scalable and can handle large datasets efficiently.
- Parallelization: The algorithm can be parallelized, making it suitable for distributed computing frameworks like Spark.
- Implicit Feedback: ALS can handle both explicit (ratings) and implicit (clicks, views) feedback.
- Cold Start Problem: ALS struggles with new users or items that have no interaction history.
- Hyperparameter Tuning: The performance of ALS depends on the choice of hyperparameters like the number of factors and regularization parameter.
The model's performance is evaluated using the RMSE metric. A lower RMSE indicates better performance. The recommendations generated by the model can be used to suggest movies to users based on their predicted ratings.
This project demonstrates how to build a recommendation system using ALS in PySpark. The model is trained on the MovieLens dataset and evaluated using RMSE. The recommendations generated by the model can be used to enhance user experience by suggesting relevant movies.
This README provides a comprehensive overview of the code and the ALS algorithm. It explains the steps involved in building a recommendation system and provides insights into the underlying principles of collaborative filtering using ALS.