Databricks-Machine-Learning-Professional Databricks Certified Machine Learning Professional Questions and Answers

A pyfunc model is a generic Python model that can be loaded and used to make predictions using any Python ML library. By logging a custom model class that implements the preprocessing logic as a pyfunc model, the same logic will be applied when loading the model and calling the predict method on new data. This ensures consistency and reproducibility of the model predictions. The fit method is not relevant for loading a logged model, as it is only used for training the model. The pyfunc model does not affect the parallelizability or the need for pipeline-like objects of the model deployment. References: mlflow.pyfunc, Creating custom Pyfunc models

To obtain run-level results for an experiment ID in a Spark DataFrame, you can use the mlflow.search_runs(exp_id) function. This function takes a filter string, which acts as a filter to the query, and returns a pandas.DataFrame of runs, where each metric, parameter, and tag are expanded into their own columns named metrics., params., and tags.* respectively1. You can then convert the pandas.DataFrame to a Spark DataFrame using the spark.createDataFrame() method2. The other options are incorrect because:

• Option A: client.list_run_infos(exp_id) will return a list of RunInfo objects, not a DataFrame3.
• Option B: spark.read.format(“delta”).load(exp_id) will not work for MLflow experiments, as it expects a Delta table path, not an experiment ID4.
• Option C: There is a way to programmatically return row-level results from an MLflow experiment using the mlflow.search_runs(exp_id) function1.
• Option E: spark.read.format(“mlflow-experiment”).load(exp_id) is not a valid format for reading MLflow experiments, as there is no such format registered in Spark5. References: Search Runs, Create a DataFrame from an RDD, a list or a pandas.DataFrame, MlflowClient.list_run_infos, Delta Lake Quickstart, spark.read.format

Jensen-Shannon (JS) distance is a symmetric measure of the distance between two probability distributions. It is based on the Kullback-Leibler (KL) divergence, but unlike KL, it does not depend on which distribution is considered the reference and which is considered the comparison. JS distance is also bounded between 0 and 1, which makes it easier to interpret and compare across different features.

Kolmogorov-Smirnov (KS) test is a non-parametric test that compares the cumulative distribution functions (CDFs) of two samples. It calculates the maximum difference between the CDFs and compares it to a critical value to determine if the samples are from the same distribution. KS test is sensitive to the shape, location, and scale of the distributions, but it can also be too sensitive for large datasets, where small changes can result in significant differences.

JS distance is more robust than KS test for numeric feature drift detection because it can handle large datasets without being too sensitive to minor fluctuations. It can also capture changes in the shape and spread of the distributions, not just the location. JS distance does not require any manual threshold or cutoff determinations, unlike KS test, which requires choosing a significance level and a critical value. JS distance can also be generalized to multivariate cases, while KS test is limited to univariate cases. References:

• Which test is the best? We compared 5 methods to detect data drift on large datasets
• The Jensen-Shannon Divergence: A Measure of Distance Between Probability Distributions
• Kullback–Leibler vs Kolmogorov-Smirnov distance

To create nested runs in MLflow, you need to specify the nested=True argument in the parent run, and omit it in the child runs. This will set a special tag (mlflow.utils.mlflow_tags.MLFLOW_PARENT_RUN_ID) in the child runs that indicates the parent run ID. This way, the UI will show the child runs under the parent run in a collapsible hierarchy. The indentation of the code blocks is not relevant, as long as the child runs are created within the context of the parent run. The run name parameter is also not relevant, as it does not affect the nesting of the runs. References: Understanding Parent and Child Runs in MLflow, Nested runs using MLflowClient, creating-child-runs-in-mlflow

The correct way to log a scikit-learn model using MLflow is to use the mlflow.sklearn.log_model function, which logs the model as an MLflow artifact for the current run1. This function takes the scikit-learn model object and an artifact path as required arguments, and optionally accepts other arguments such as conda environment, code paths, serialization format, etc2. The other options are incorrect because they either use the wrong flavor (spark instead of sklearn), the wrong function (load_model or track_model instead of log_model), or do not exist (track_model is not a valid MLflow function). References: MLflow Models, mlflow.sklearn

Batch deployment is the most common paradigm for machine learning projects, as it accounts for 80-90% of deployments1. Batch deployment leverages databases and object storage to process a large batch of input data at once, rather than processing each input data point individually in real time. Batch deployment is well suited for large data volume scenarios that need efficient processing but response time isn’t critical. For example, you might use batch deployment to generate recommendations for customers based on their purchase history, or to detect anomalies in sensor data collected over a period of time. Batch deployment is less expensive and more efficient than real-time deployment, as the underlying compute runs only during the batch job and can be scaled up or down as needed. Batch deployment can be implemented using Databricks Machine Learning endpoints, which provide a unified interface to invoke and manage model deployments across compute types2. References:

• Databricks Machine Learning Professional Exam Guide, Section 3: Model Deployment, p. 4
• Databricks Machine Learning Documentation, Deploying ML Models, Batch Inference, p. 1
• Machine Learning in Production - Databricks, The Four Deployment Paradigms, p. 9
• Deploy machine learning models in production environments, Deployment methods, p. 1

Concept drift is a change in the relationship between input and output data over time, which can affect the performance and accuracy of the model. To monitor concept drift, one needs to compare the predicted values from the model with the observed (actual) label values from the data source. A common way to do this is to compute an evaluation metric, such as accuracy, precision, recall, F1-score, etc., using the observed and predicted values. This metric can indicate how well the model is performing on the current data, and how it differs from the baseline performance on the training data. If the metric shows a significant decline over time, it may suggest that concept drift has occurred and the model needs to be updated123 References:

• Monitoring Concept Drift - Databricks
• Data drift, concept drift, and how to monitor for them - Mona Labs
• 8 Concept Drift Detection Methods To Use With ML Models - Aporia

Batch serving is a type of model deployment where predictions are computed in advance and stored for later use. This can be useful when the feature values are known ahead of time and the predictions need to be delivered quickly. Querying stored predictions can be faster than computing predictions in real-time, especially if the model is complex or the data volume is high. Real-time serving is a type of model deployment where predictions are computed on-demand at query time. This can be useful when the feature values are not known in advance or the predictions need to be up-to-date. However, computing predictions in real-time can be slower than querying stored predictions, especially if the model is simple or the data volume is low. References: [Batch Serving], [Real-time Serving] Message has links.

To log feature importance data from a CSV file as an artifact of an MLflow run, the correct method is mlflow.log_artifact, which takes the local file path and the artifact path as arguments.The mlflow.log_model method is used to log a model with a specific flavor, such as pyfunc or spark. The mlflow.log_data method does not exist. The mlflow.log_model_and_data method also does not exist, and it is not necessary to log both the model and the data in this case. References:

• MLflow Tracking API
• MLflow Models

The Databricks REST APIs allow users to programmatically access Databricks features and services, such as clusters, jobs, workspaces, libraries, secrets, and more. The Jobs API enables users to create, edit, delete, run, and list jobs, as well as manage job runs. Users can use the Jobs API to programmatically create a new Databricks job whose schedule depends on the result of some automated tests in a machine learning pipeline. For example, users can use the create endpoint to create a new job with a cron_schedule parameter that specifies the cron expression for the job schedule. Users can also use the run_now endpoint to run an existing job with a run_name parameter that specifies the name of the run. Users can use the runs/get endpoint to get the run result state of a job run, and use conditional logic to trigger another job run based on the result. Users can also use the runs/list endpoint to list all the runs of a job, and use the runs/cancel endpoint to cancel a job run if needed. References:

• Databricks REST API — Databricks Documentation, p. 1
• Jobs API — Databricks Documentation, p. 1-2
• Create and run Databricks Jobs | Databricks on AWS, Create and run jobs using the CLI, API, or notebooks, p. 1

To compute predictions for spark_df when the missing feature values can be found in the Feature Store by searching for features by customer_id, you can use the following code block:

Python

# Get the missing features from the Feature Store using the model URI and the customer_id column

df = fs.get_missing_features(spark_df, model_uri, lookup_key="customer_id")

# Score the DataFrame using the model URI and the Feature Store Client

fs.score_batch(model_uri, df)

AI-generated code. Review and use carefully. More info on FAQ.

The fs.get_missing_features method takes a Spark DataFrame, a model URI, and a lookup key as arguments. It returns a new Spark DataFrame that contains the originalcolumns plus the missing features that are required by the model. The missing features are retrieved from the Feature Store by joining the DataFrame with the feature tables using the lookup key. The lookup key must match the primary key of the feature tables. The model URI must point to a registered model that was trained using features from the Feature Store1.

The fs.score_batch method takes a model URI and a Spark DataFrame as arguments. It applies the model to the DataFrame and returns a new Spark DataFrame that contains the original columns plus a prediction column. The model URI must point to a registered model that was trained using features from the Feature Store2.

The other options are incorrect because:

• Option A: fs.score_model is not a valid method name, as it is missing an underscore. The correct method name is fs.score_batch2.
• Option B: fs.score_model without getting the missing features will not work, as the model expects the DataFrame to have all the features that were used for training. The correct way is to use fs.get_missing_features before fs.score_batch12.
• Option D: fs.score_batch without getting the missing features will not work, as the model expects the DataFrame to have all the features that were used for training. The correct way is to use fs.get_missing_features before fs.score_batch12.
• Option E: fs.score_batch without specifying the lookup key will not work, as the fs.get_missing_features method requires a lookup key to join the DataFrame with the feature tables. The correct way is to use fs.get_missing_features with the lookup key “customer_id” before fs.score_batch12. References: Get missing features, Score batch

 To read data from a stream source, such as Kafka, socket, or rate, the spark.readStream method should be used instead of spark.read. The spark.readStream method returns a streaming DataFrame that represents the unbounded input data stream. The spark.readStream method supports the same options and formats as the spark.read method, such as schema, delta, csv, json, etc. The spark.readStream method can also read from a Delta table as a stream source, by specifying the format("delta") and the path or table name of the Delta table123

The other options are incorrect because:

• A. Replacing “inference” with the path to the location of the Delta table does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the path or table name of the Delta table can be specified as an option or argument.
• B. Replacing schema(schema) with option("maxFilesPerTrigger", 1) does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the schema can be specified as an option or argument. The option("maxFilesPerTrigger", 1) is an optional configuration that limits the number of files processed in each trigger for file-based stream sources, such as delta, csv, json, etc. It does not affect the reading of data from a stream source4
• D. Replacing format("delta") with format("stream") does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the format can be specified as an option or argument. The format("stream") is not a valid format for reading data from a stream source. The supported formats are delta, kafka, socket, rate, etc1
• E. Replacing predict with a stream-friendly prediction function does not change the fact that spark.read is used to read from a stream source, which is not supported. The spark.readStream method should be used instead, and the prediction function can be applied to the streaming DataFrame as usual. The predict function does not need to be changed, as long as it can accept a streaming DataFrame as input and return a column of predictions as output5

References:

• Input Sources - Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Structured Streaming + Delta Lake - Databricks
• Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Configuration - Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Machine Learning with Structured Streaming - Databricks

Concept drift in machine learning and data mining refers to the change in the relationships between input and output data in the underlying problem over time. In other domains, this change maybe called “covariate shift,” “dataset shift,” or “nonstationarity.” Concept drift can affect the performance and accuracy of predictive models that assume a static relationship between input and output variables. Concept drift can be caused by various factors, such as changes in user behavior, environmental conditions, market trends, etc. Concept drift can be detected and handled by various methods, such as periodic retraining, online learning, ensemble methods, etc References:

• Concept drift - Wikipedia
• A Gentle Introduction to Concept Drift in Machine Learning
• Model Drift & Machine Learning: Concept Drift, Feature Drift, Etc.
• Data Drift vs. Concept Drift: What Is the Difference?

Label drift is a type of data drift that occurs when there is a change in the probability of a label P(Y). This means that the output distribution of the model’s data source has changed over time, which can affect the performance and accuracy of the model. For example, if a model is trained to classify images of animals, and the data source starts to provide more images of cats than dogs, this would be an example of label drift. Label drift can be caused by various factors, such as changes in user behavior, environmental conditions, market trends, etc. Label drift can be detected and handled by various methods, such as periodic retraining, online learning, ensemble methods, etc123 References:

• Monitoring Feature Drift - Databricks
• Drift Metrics: How to Select the Right Metric to Analyze Drift
• Detect data drift on datasets (preview) - Azure Machine Learning

Categorical feature drift is a change in the distribution of the input data over time for categorical features, which can affect the performance and accuracy of the model. Monitoring categorical feature drift is important to ensure that the model is still valid and reliable for the current data. One simple statistic to monitor for categorical feature drift is the combination of mode, number of unique values, and percentage of missing values for each categorical feature. These statistics can provide a quick overview of the changes in the data distribution, such as the most frequent category, the diversity of categories, and the quality of data. If these statistics deviate significantly from the baseline values, it may indicate a categorical feature drift. However, these statistics may not capture all the nuances of the data distribution, such as the relative frequencies of different categories, the similarity of categories, etc. Therefore, other methods, such as statistical tests or information-theoretic measures, may be needed to complement the simple statistics and provide a more comprehensive analysis of the categorical feature drift123 References:

• Monitoring Feature Drift - Databricks
• Drift Metrics: How to Select the Right Metric to Analyze Drift
• Detect data drift on datasets (preview) - Azure Machine Learning