Overview

State of Affairs at Netflix

1. Astyanax recipes - these allow users to perform complex operations such as reading all rows from multiple token ranges, chunked object storage, optimistic distributed locking etc.
2. Entity Mapper - this layer allows users to easily map their business logic entities to persistent objects in Cassandra.

DataStax Java Driver

1. Out-of-the-box support for async operations over netty
2. Cursors for pagination
3. Query tracing when debugging
4. Support for collections
5. Built-in load balancing, retry policies etc.

• schema-in-cassandra-1-1
• whats-new-in-cql-3-0
• cql3-for-cassandra-experts
• thrift-to-cql3

1.0 and 2.0 releases

Integration

Structured api with async support

Astyanax helps with prepared statement management

Astyanax provides a unified interface across multiple drivers

JD 1.0 v/s 2.0

Migration path

Caveats / Findings

Performance characteristics

Basic setup

1. The standard Astyanax client that uses the thrift driver underneath
2. The standard Java Driver released by DataStax.
3. Astyanax over Java Driver - a limited implementation of the Astyanax interface that uses the Java Driver underneath instead of thrift.

Batch Mutations

There is about a 5-10% deviation between thrift and the java driver, where thrift was faster than java driver.

Single Row Reads

Variation between reads was lesser (2%-5%).

Using Prepared Statements

Regular query

prepared statement

As you can see below, the performance difference between these 2 blocks of code is significant, hence users are encouraged on using prepared statements when using either Java Driver or the Astyanax adaptor over Java Driver.

Writes

The catch with prepared-statements management!

An important caveat to note here is that prepared statements work well when your queries are highly cacheable, which means that your queries have the same signature.  For writes this means that if you are adding, updating, deleting columns with different timestamps and TTLs then you can't leverage prepared statements since there is no "cacheability" in those queries. If you want the mutation batches to be highly cacheable, then you must use the same TTLs and timestamps when re-using prepared statements for subsequent batches.

Similarly for reads, you can't prepare a statement by doing a row query and then reuse that statement for a subsequent row query that also has a column slice component to it, since the underlying query structure is actually different.

Here is an example to illustrate my point

// Select a row with a column slice specification (column range query) select * from test1 where key = ? and column1 > ?; // is very different from a similar column slice query select * from test1 where key = ? and column1 > ? and column1 < ?; // Hence both these queries have different signatures and hence need their own prepared statements. Hence use the automatic statement management feature in Astyanax with caution. Inspect your table schema and your query patterns. When using queries with different patterns, turn caching OFF. Rows v/s Columns In CQL3 columns can be rows as well depending on your column family schema. Here is a simple example. Say that you have a simple table with the following schema. Key Validator: INT_TYPE Column Comparator: INT_TYPE Default Validator : UTF8_TYPE Here is what data would look like at the storage level (example data)

But in CQL3 the same CF has a clustering key that corresponds to your column comparator. Hence unique columns will appear as distinct rows.

Summary - Astyanax in 2014

It is our pleasure to release PigPen to the world today. PigPen is map-reduce for Clojure. It compiles to Apache Pig, but you don’t need to know much about Pig to use it.

What is PigPen?

• A map-reduce language that looks and behaves like clojure.core
• The ability to write map-reduce queries as programs, not scripts
• Strong support for unit tests and iterative development

Note: If you are not familiar at all with Clojure, we strongly recommend that you try a tutorial here, here, or here to understand some of the basics.

Really, yet another map-reduce language?

If you know Clojure, you already know PigPen

The primary goal of PigPen is to take language out of the equation. PigPen operators are designed to be as close as possible to the Clojure equivalents. There are no special user defined functions (UDFs). Define Clojure functions, anonymously or named, and use them like you would in any Clojure program.

Here’s the proverbial word count:

(require '[pigpen.core :as pig])

(defn word-count [lines]
  (->> lines
    (pig/mapcat #(-> % first
                   (clojure.string/lower-case)
                   (clojure.string/replace #"[^\w\s]" "")
                   (clojure.string/split #"\s+")))
    (pig/group-by identity)
    (pig/map (fn [[word occurrences]] [word (count occurrences)]))))

This defines a function that returns a PigPen query expression. The query takes a sequence of lines and returns the frequency that each word appears. As you can see, this is just the word count logic. We don’t have to conflate external concerns, like where our data is coming from or going to.

Will it compose?

Yep - PigPen queries are written as function compositions - data in, data out. Write it once and avoid the copy & paste routine.

Here we use our word-count function (defined above), along with a load and store command, to make a PigPen query:

(defn word-count-query [input output]
  (->>
    (pig/load-tsv input)
    (word-count)
    (pig/store-tsv output)))

This function returns the PigPen representation of the query. By itself, it won’t do anything - we have to execute it locally or generate a script (more on that later).

You like unit tests? Yeah, we do that

With PigPen, you can mock input data and write a unit test for your query. No more crossing your fingers & wondering what will happen when you submit to the cluster. No more separate files for test input & output.

Mocking data is really easy. With pig/return and pig/constantly, you can inject arbitrary data as a starting point for your script.

A common pattern is to use pig/take to sample a few rows of the actual source data. Wrap the result with pig/return and you’ve got mock data.

(use 'clojure.test)

(deftest test-word-count
  (let [data (pig/return [["The fox jumped over the dog."]
                          ["The cow jumped over the moon."]])]
    (is (= (pig/dump (word-count data))
           [["moon" 1]
            ["jumped" 2]
            ["dog" 1]
            ["over" 2]
            ["cow" 1]
            ["fox" 1]
            ["the" 4]]))))

The pig/dump operator runs the query locally.

Closures (yes, the kind with an S)

Parameterizing your query is trivial. Any in-scope function parameters or let bindings are available to use in functions.

(defn reusable-fn [lower-bound data]
  (let [upper-bound (+ lower-bound 10)]
    (pig/filter (fn [x] (< lower-bound x upper-bound)) data)))

Note that lower-bound and upper-bound are present when we generate the script, and are made available when the function is executed within the cluster.

So how do I use it?

Just tell PigPen where to write the query as a Pig script:

(pig/write-script "word-count.pig"
                  (word-count-query "input.tsv" "output.tsv"))

And then you have a Pig script which you can submit to your cluster. The script uses pigpen.jar, an uberjar with all of the dependencies, so make sure that is submitted with it. Another option is to build an uberjar for your project and submit that instead. Just rename it prior to submission. Check out the tutorial for how to build an uberjar.

As you saw before, we can also use pig/dump to run the query locally and return Clojure data:

=> (def data (pig/return [["The fox jumped over the dog."]
                          ["The cow jumped over the moon."]]))
#'pigpen-demo/data

=> (pig/dump (word-count data))
[["moon" 1] ["jumped" 2] ["dog" 1] ["over" 2] ["cow" 1] ["fox" 1] ["the" 4]]

If you want to get started now, check out getting started & tutorials.

Why do I need Map-Reduce?

Map-Reduce is useful for processing data that won’t fit on a single machine. With PigPen, you can process massive amounts of data in a manner that mimics working with data locally. Map-Reduce accomplishes this by distributing the data across potentially thousands of nodes in a cluster. Each of those nodes can process a small amount of the data, all in parallel, and accomplish the task much faster than a single node alone. Operations such as join and group, which require coordination across the dataset, are computed by partitioning the data with a common join key. Each value of the join key is then sent to a specific machine. Once that machine has all of the potential values, it can then compute the join or do other interesting work with them.

To see how PigPen does joins, let’s take a look at pig/cogroup. Cogroup takes an arbitrary number of data sets and groups them all by a common key. Say we have data that looks like this:

foo:

  {:id 1, :a "abc"}
  {:id 1, :a "def"}
  {:id 2, :a "abc"}

bar:

  [1 42]
  [2 37]
  [2 3.14]

baz:

  {:my_id "1", :c [1 2 3]]}

If we want to group all of these by id, it looks like this:

(pig/cogroup (foo by :id)
             (bar by first)
             (baz by #(-> % :my_id Long/valueOf))
             (fn [id foos bars bazs] ...))

The first three arguments are the datasets to join. Each one specifies a function that is applied to the source data to select the key. The last argument is a function that combines the resulting groups. In our example, it would be called twice, with these arguments:

[1 ({:id 1, :a "abc"}, {:id 1, :a "def"})
   ([1 42])
   ({:my_id "1", :c [1 2 3]]})]
[2 ({:id 2, :a "abc"})
   ([2 37] [2 3.14])
   ()]

As you can see, this consolidates all of the values with an id of 1, all of the values with 2, etc. Each different key value can then be processed independently on different machines. By default, keys are not required to be present in all sources, but there are options that can make them required.

Hadoop provides a very low-level interface for doing map-reduce jobs, but it’s also very limited. It can only run a single map-reduce pair at a time as it has no concept of data flow or a complex query. Pig adds a layer of abstraction on top of Hadoop, but at the end of the day, it’s still a scripting language. You are still required to use UDFs (user defined functions) to do interesting tasks with your data. PigPen furthers that abstraction by making map-reduce available as a first class language.

If you are new to map-reduce, we encourage you to learn more here.

Motivations for creating PigPen

• Code reuse. We want to be able to define a piece of logic once, parameterize it, and reuse it for many different jobs.
• Consolidate our code. We don’t like switching between a script and a UDF written in different languages. We don’t want to think about mapping between differing data types in different languages.
• Organize our code. We want our code in multiple files, organized how we want - not dictated by the job it belongs to.
• Unit testing. We want our sample data inline with our unit tests. We want our unit tests to test the business logic of our jobs without complications of loading or storing data.
• Fast iteration. We want to be able to trivially inject mock data at any point in our jobs. We want to be able to quickly test a query without waiting for a JVM to start.
• Name what you want to. Most map-reduce languages force too many names and schemas on intermediate products. This can make it really difficult to mock data and test isolated portions of jobs. We want to organize and name our business logic as we see fit - not as dictated by the language.
• We’re done writing scripts; we’re ready to start programming!

Note: PigPen is not a Clojure wrapper for writing Pig scripts you can hand edit. While it’s entirely possible, the resulting scripts are not intended for human consumption.

Design & Features

PigPen was designed to match Clojure as closely as possible. Map-reduce is functional programming, so why not use an awesome functional programming language that already exists? Not only is there a lower learning curve, but most of the concepts translate very easily to big data.

In PigPen, queries are manipulated as expression trees. Each operation is represented as a map of information about what behavior is desired. These maps can be nested together to build a tree representation of a complex query. Each command also contains references to its ancestor commands. When executed, that query tree is converted into a directed acyclic query graph. This allows for easy merging of duplicate commands, optimizing sequences of related commands, and instrumenting the query with debug information.

Optimization

De-duping

When we represent our query as a graph of operations, de-duping them is trivial. Clojure provides value-equality, meaning that if two objects have the same content, they are equal. If any two operations have the same representation, then they are in fact identical. No care has to be taken to avoid duplicating commands when writing the query - they’re all optimized before executing it.

For example, say we have the following query:

(let [even-squares (->>
                     (pig/load-clj "input.clj")
                     (pig/map (fn [x] (* x x)))
                     (pig/filter even?)
                     (pig/store-clj "even-squares.clj"))
      odd-squares (->>
                    (pig/load-clj "input.clj")
                    (pig/map (fn [x] (* x x)))
                    (pig/filter odd?)
                    (pig/store-clj "odd-squares.clj"))]
  (pig/script even-squares odd-squares))

In this query, we load data from a file, compute the square of each number, and then split it into even and odd numbers. Here’s what a graph of this operation would look like:

This matches our query, but it’s doing some extra work. It’s loading the same input.clj file twice and computing the squares of all of the numbers twice. This might not seem like a lot of work, but when you do it on a lot of data, simple operations really add up. To optimize this query, we look for operations that are identical. At first glance it looks like our operation to compute squares might be a good candidate, but they actually have different parents so we can’t merge them yet. We can, however, merge the load functions because they don’t have any parents and they load the same file.

Now our graph looks like this:

Now we’re loading the data once, which will save some time, but we’re still doing the squares computation twice. Since we now have a single load command, our map operations are now identical and can be merged:

Now we have an optimized query where each operation is unique. Because we always merge commands one at a time, we know that we’re not going to change the logic of the query. You can easily generate queries within loops without worrying about duplicated execution - PigPen will only execute the unique parts of the query.

Serialization

After we’re done processing data in Clojure, our data must be serialized into a binary blob so Pig can move it around between machines in a cluster. This is an expensive, but essential step for PigPen. Luckily, many consecutive operations in a script can often be packed together into a single operation. This saves a lot of time by not serializing and deserializing the data when we don’t need to. For example, any consecutive map, filter, and mapcat operations can be re-written as a single mapcat operation.

Let’s look at some examples to illustrate this:

In this example, we start with a serialized (blue) value, 4, deserialize it (orange), apply our map function, and re-serialize the value.

Now let’s try a slightly more complex (and realistic) example. In this example, we apply a map, mapcat, and filter operation.

If you haven’t used it before, mapcat is an operation where we apply a function to a value and that function returns a sequence of values. That sequence is then ‘flattened’ and each single value is fed into the next step. In Clojure, it’s the result of combining map and concat. In Scala, this is called flatMap and in c# it’s called selectMany.

In the diagram below, the flow on the left is our query before the optimization; the right is after the optimization. We start with the same value, 4, and calculate the square of the value; same as the first example. Then we take our value and apply a function that decrements the value, returns the value, and increments the value. Pig then takes this set of values and flattens them, making each one an input to the next step. Note that we had to serialize and deserialize the data when interacting with Pig. The third and final step is to filter the data; in this example we’re retaining only odd values. As you can see, we end up serializing and deserializing the data in between each step.

The right hand side shows the result of the optimization. Put simply, each operation now returns a sequence of elements. Our map operation returns a sequence of one element, 16, our mapcat remains the same, and our filter returns a sequence of zero or one elements. By making these commands more uniform, we can merge them more easily. We end up flattening more sequences of values within the set of commands, but there is no serialization cost between steps. While it looks more complex, this optimization results in much faster execution of each if these steps.

Testing, Local Execution, and Debugging

Iterative development, testing, and debuggability are key tenants of PigPen. When you have jobs that can run for days at a time, the last thing you need is an unexpected bug to show up in the eleventh hour. PigPen has a local execution mode that’s powered by rx. This allows us to write unit tests for our queries. We can then know with very high confidence that something will not crash when run and will actually return the expected results. Even better, this feature allows for iterative development of queries.

Typically, we start with just a few records of the source data and use that to populate a unit test. Because PigPen returns data in the REPL, we don’t have to go elsewhere to build our test data. Then, using the REPL, we add commands to map, filter, join, and reduce the mock data as required; each step of the way verifying that the result is what we expect. This approach produces more reliable results than building a giant monolithic script and crossing your fingers. Another useful pattern is to break up large queries into smaller functional units. Map-reduce queries tend to explode and contract the source data by orders of magnitude. When you try to test the script as a whole, you often have to start with a very large amount of data to end up with just a few rows. By breaking the query into smaller parts, you can test the first part, which may take 100 rows to produce two; and then test the second part by using those two rows as a template to simulate 100 more fake ones.

Debug mode has proven to be really useful for fixing the unexpected. When enabled, it will write to disk the result of every operation in the script, in addition to the normal outputs. This is very useful in an environment such as Hadoop, where you can’t step through code and hours may pass in between operations. Debug mode can also be coupled with a graph-viz visualization of the execution graph. You can then visually associate what it plans to do with the actual output of each operation.

To enable debug mode, see the options for pig/write-script and pig/generate-script. It will write the extra debug output to the folder specified.

Example of debug mode enabled:

(pig/write-script {:debug "/debug-output/"} "my-script.pig" my-pigpen-query)

To enable visualization, take a look at pig/show and pig/dump&show

Example of visualization:

(pig/show my-pigpen-query)        ;; Shows a graph of the query
(pig/dump&show my-pigpen-query)   ;; Shows a graph and runs it locally

Extending PigPen

One nice feature of PigPen is that it’s easy to build your own operators. For example, we built set and multi-set operators such as difference and intersection. These are just variants of other operations like co-group, but it’s really nice to define them once, test them thoroughly, and not have to think about the logic behind a multiset intersection of n sets ever again.

This is useful for more complex operators as well. We have a reusable statistics operator that computes the sum, avg, min, max, sd, and quantiles for a set of data. We also have a pivot operator that groups dimensional fields in the data and counts each group.

While each of these by themselves are simple operations, when you abstract them out of your query, your query starts to become a lot smaller and simpler. When your query is smaller and simpler, you can spend more time focusing on the actual logic of the problem you’re trying to solve instead of re-writing basic statistics each time. What you’re doing becomes clearer, every step of the way.

Why Pig?

We chose Pig because we didn’t want to re-implement all of the optimization work that has gone into Pig already. If you take away the language, Pig does an excellent job of moving big data around. Our strategy was to use Pig’s DataByteArray binary format to move around serialized Clojure data. In most cases, we found that Pig didn’t need to be aware of the underlying types present in the data. Byte arrays can be compared trivially and quickly, so for joins and groupings, Pig simply needs to compare the serialized blob. We get Clojure’s great value equality for free as equivalent data structures produce the same serialized output. Unfortunately, this doesn’t hold true for sorting data. The sorted order of a binary blob is far less than useful, and doesn’t match the sorted order of the native data. To sort data, we must fall back to the host language, and as such, we can only sort on simple types. This is one of very few places where Pig has imposed a limitation on PigPen.

We did evaluate other languages before deciding to build PigPen. The first requirement was that it was an actual programming language, not just a scripting language with UDFs. We briefly evaluated Scalding, which looks very promising, but our team primarily uses Clojure. It could be said that PigPen is to Clojure what Scalding is to Scala. Cascalog is usually the go-to language for map-reduce in Clojure, but from past experiences, datalog has proven less than useful for everyday tasks. There’s a complicated new syntax and concepts to learn, aligning variable names to do implicit joins is not always ideal, misplaced ordering of operations can often cause big performance problems, datalog will flatten data structures (which can be wasteful), and composition can be a mind bender.

We also evaluated a few options to use as a host language for PigPen. It would be possible to build a similar abstraction on top of Hive, but schematizing every intermediate product doesn’t fit well with the Clojure ideology. Also, Hive is similar to SQL, making translation from a functional language more difficult. There’s an impedance mismatch between relational models like SQL and Hive and functional models like Clojure or Pig. In the end, the most straightforward solution was to write an abstraction over Pig.

Future Work

Currently you can reference in-scope local variables within code that is executed remotely, as shown above. One limitation to this feature is that the value must be serializable. This has the downside of not being able to utilize compiled functions - you can’t get back the source code that created them in the first place. This means that the following won’t work:

(defn foo [x] ...)

(pig/map foo)

In this situation, the compiler will inform you that it doesn’t recognize foo. We’re playing around with different methods for requiring code remotely, but there are some nuances to this problem. Blindly loading the code that was present when the script was generated is an easy option, but it might not be ideal if that code accidentally runs something that was only intended to run locally. Another option would be for the user to explicitly specify what to load remotely, but this poses challenges as well, such as an elegant syntax to express what should be loaded. Everything we’ve tried so far is a little clunky and jar hell with Hadoop doesn’t make it any easier. That said, any code that’s available can be loaded from within any user function. If you upload your uberjar, you can then use a require statement to load other arbitrary code.

So far, performance in PigPen doesn’t seem to be an issue. Long term, if performance issues crop up, it will be relatively easy to migrate to running PigPen directly on Hadoop (or similar) without changing the abstraction. One of the key performance features we still have yet to build is incremental aggregation. Pig refers to this as algebraic operators (also referenced by Rich Hickey here as combining functions). These are operations that can compute partial intermediate products over aggregations. For example, say we want to take the average of a LOT of numbers - so many that we need map-reduce. The naive approach would be to move all of the numbers to one machine and compute the average. A better approach would be to partition the numbers, compute the sum and count of each of these smaller sets, and then use those intermediate products to compute the final average. The challenge for PigPen will be to consume many of these operations within a single function. For example, say we have a set of numbers and we want to compute the count, sum, and average. Ideally, we would want to define each of these computations independently as algebraic operations and then use them together over the same set of data, having PigPen do the work of maintaining a set of intermediate products. Effectively, we need to be able to compose and combine these operations while retaining their efficiency.

We use a number of other Pig & Hadoop tools at Netflix that will pair nicely with PigPen. We have some prototypes for integration with Genie, which adds a pig/submit operator. There’s also a loader for Aegisthus data in the works. And PigPen works with Lipstick as the resulting scripts are Pig scripts.

Conclusion

PigPen has been a lot of fun to build and we hope it’s even more fun to use. For more information on getting started with PigPen and some tutorials, check out the tutorial, or to contribute, take a look at our Github page: https://github.com/Netflix/PigPen

There are three distinct audiences for PigPen, so we wrote three different tutorials:

• Those coming from the Clojure community who want to do map-reduce: PigPen for Clojure users
• Those coming from the Pig community who want to use Clojure: PigPen for Pig users
• And a general tutorial for anybody wanting to learn it all: PigPen Tutorial

If you know both Clojure and Pig, you’ll probably find all of the tutorials interesting.

The full API documentation is located here

And if you love big data, check out our jobs.

In previous posts, we discussed how the Hadoop platform at Netflix leverages AWS’s S3 offering (read more here). In short, Netflix considers S3 the “source of truth” for all data warehousing.  There are many attractive features that draw us to this service including: 99.999999999% durability, 99.99% availability, effectively infinite storage, versioning (data recovery), and ubiquitous access. In combination with AWS’s EMR, we can dynamically expand/shrink clusters, provision/decommission clusters based on need or availability of reserved capacity, perform live cluster swapping without interrupting processing, and explore new technologies all utilizing the same data warehouse.  In order to provide the capabilities listed above, S3 makes one particular concession which is the focus of this discussion: consistency.

The consistency guarantees for S3 vary by region and operation (details here), but in general, any list or read operation is susceptible to inconsistent information depending on preceding operations. For basic data archival, consistency is not a concern.  However, in a data centric computing environment where information flows through a complex workflow of computations and transformations, an eventually consistent model can cause problems ranging from insidious data loss to catastrophic job failure.

How Inconsistency Impacts Processing

The Netflix ETL Process is predominantly Pig and Hive jobs scheduled through enterprise workflow software that resolves dependencies and manages task execution.  To understand how eventual consistency affects processing, we can distill the process down to a simple example of two jobs where the results of one feed into another.  If we take a look at Pig-1 from the diagram, it consists of two MapReduce jobs in a pipeline.  The initial dataset is loaded from S3 due to the source location referencing an S3 path. All intermediate data is stored in HDFS since that is the default file system.  Consistency is not a concern for these intermediate stages.  However, the results from Pig-1 are stored directly back to S3 so the information is immediately available for any other job across all clusters to consume.

Pig-2 is activated based on the completion of Pig-1 and immediately lists the output directories of the previous task.  If the S3 listing is incomplete when the second job starts, it will proceed with incomplete data.  This is particularly problematic, as we stated earlier, because there is no indication that a problem occurred.  The integrity of resulting data is entirely at the mercy of how consistent the S3 listing was when the second job started.

A variety of other scenarios may result in consistency issues, but inconsistent listing is our primary concern.  If the input data is incomplete, there is no indication anything is wrong with the result.  Obviously it is noticeable when the expected results vary significantly from long standing patterns or emit no data at all, but if only a small portion of input is missing the results will appear convincing.  Data loss occurring at the beginning of a pipeline will have a cascading effect where the end product is wildly inaccurate.  Due to the potential impact, it is essential to understand the risks and methods to mitigate loss of data integrity.

Approaches to Managing Consistency

The Impractical

Staging Data

Consistency through Convention

Conventions can be used to eliminate some cases of inconsistency.  Read and list inconsistency resulting from overwriting the same location can result in data corruption in that a listing may include old versions of data with new therefore producing an amalgam of two incomplete datasets.  Eliminating update inconsistency is achievable by imposing a convention where the same location is never overwritten.  Here at Netflix, we encourage the use of a batching pattern, where results are written into partitioned batches and the Hive metastore only references the valid batches.  This approach removes the possibility of inconsistency due to update or delete.  For all AWS regions except US Standard that provide “read-after-write” consistency, this approach may be sufficient, but relies on strict adherence.

Secondary Index

What makes s3mper a hybrid approach is its use of the S3 listing for comparison and only maintaining a window of consistency.  The “source of truth” is still S3, but with an additional layer of checking added.  The window of consistency allows for falling back to the S3 listing without concern that the secondary index will fail and lose important information or risk consistency issues that arise from using tools outside the hadoop stack to modify data in S3.

• Recovery: When an inconsistent listing is identified, s3mper will optionally delay the listing and retry until consistency is achieved.  This will delay a job only long enough for data to become available without unnecessarily impacting job performance.
• Notification: If listing cannot be achieved, a notification is sent immediately and a determination can be made as to whether to kill the job or let it proceed with incomplete data.
• Reporting: A variety of events are sent to track the number of recoveries, files missed, what jobs were affected, etc.
• Configurability:  Options are provided to control how long a job should wait, how frequently to recheck a listing, and whether to fail a job if the listing is inconsistent.
• Modularity: The implementations for the metastore and notifications can be overridden based on the environment and services at your disposal.
• Administration: Utilities are provided for inspecting the metastore and resolving conflicts between the secondary index in DynamoDB and the S3 index.

ByRanjit Mavinkurve,Justin Becker andBen Christensen

Extending Our Current Insight Capabilities

Context

The New Way

Timeliness Matters

Event Stream Processing

A Picture is Worth a Thousand Words

Summary

Join Us!

Design of Scryer

Scryer has a simple data flow architecture. On a very high level, historical data flows into Scryer, and predicted actions flow out. The diagram below shows the architecture.

• Emitting predictions and action steps to our monitoring dashboard at scheduled time. This is great for simulating the behavior of Scryer. We can easily visualize the predictions and actions, and compare the predictions with the actual workload in a same graph.
• Scheduling each step using AWS API for Scheduled Actions
• Scheduling actions that will scale a cluster using EC2 API

Metrics for Prediction Algorithms

• It has a clear, relatively stable, and preferably a recurring pattern. We can predict reliably only what has repeatedly happened in the past.
• It is independent of cluster performance. We deploy our code frequently, and the performance of a deployment may vary per deployment. If the metrics depends on cluster performance, prediction may deviate widely from the actual values of the metrics.

Once we determined which metrics to predict on, we would also need to figure out how to calculate scaling actions. Since the goal of auto scaling is to ensure a cluster has sufficient number of machines to serve all the user traffic, all we need to do is to predict the size of cluster, which depends on the average throughput of a server:

Prediction Algorithms

• They have clear weekly periodicity for the same day of a week. That is, the traffic of two adjacent Tuesdays is more similar than that of adjacent Tuesday and Wednesday.
• Their daily patterns are similar, albeit different in shapes and scales
• They have some small spikes and drops that we can deem as noise.
• The change of traffic is relatively constant week by week. In other words, the traffic at the same time in the same day moves approximately linearly week by week.
• There could be occasional large spikes or large drops due to system outages.

FFT-Based Prediction

The FFT based algorithm is also capable of ignoring outages. It detects outages by applying standard statistical methods. Once an outage is detected, the algorithm will iteratively apply FFT on adjusted data until the outage is ignored. The following figure shows that a simulated big drop is reasonably ignored: 

The prediction algorithm undergoes multiple iterations to gradually remove the effect of such drop, as shown by the figure below. The first iteration is red, and the last iteration is yellow. We can see that prediction becomes better with each iteration. 

Linear Regression on Clustered Data Points

This diagram below shows workload of one of Netflix clusters within a 30-minute window. The workload does not fluctuate more than 4%. 

Once we have clusters of data points, we can then apply linear regression. The blue dots are selected clustered points, and the red line is the result of linear regression. Once we obtain the line, we can predict by a simple extrapolation of the line to a future time.

Future Work

• Making Scryer distributed. The current implementation of Scryer runs on a single server. It is capable of handling hundreds of clusters, and tolerates temporary system crash because it checkpoints important states in Cassandra. That said, making it distributed can reduce Scryer’s bootstrap time, therefore reducing its potential down time. A distributed Scryer can also be scaled up to handle many more clusters. Distributing Scryer also helps its resilience. If the single instance fails, so does Scryer. Of course, we still have AAS, but that is not optimal. Plus, because it is on a single instance, we have to opt it out of Chaos Monkey. Opting it in gives us more data on how the system fares if it does drop out, so being distributed has that benefit as well.
• Implementing automatic feedback loop so Scryer can auto tune. We record and monitor the accuracy of Scryer’s predictions, effect of scaled actions, as well instance start time. We then use such data to tune parameters of our algorithms. This work, however, can be largely automated. We plan to implement a trend detector. If the prediction starts to consistently deviate from actual workload, the detector will capture such a deviation, and feed it to an auto-correcting module. The auto-correcting module will compensate the auto scaling accordingly, and will also tune the prediction algorithm if needed.  
• Improving our prediction algorithms. For example, we ran experiments to find out how to choose cluster of data points for linear regression. We plan to automate this process so we always get accurate parameters for choosing data points. We also plan to improve the heuristics on how to filter out noises in our FFT-based algorithms.

Conclusion

Distributing Machine Learning: At what level?

Optimizing the CUDA Kernel

PCI configuration space and virtualized environments

G2 Instances

Distributed Bayesian Hyperparameter Optimization

Conclusions

by Joel Sass

In the Spring of 2013, the User Experience team was gearing up for the impending Netherlands country launch scheduled for September. To reduce barriers to adoption, we wanted to launch with a smooth signup experience on smartphones and tablets. Additionally, we were planning to implement the iDEAL online payment method, which is commonly used in the Netherlands but new to us both technically and from a user experience perspective.

At that time, we had two very different technology stacks that served our signup experience: one for desktop browsers and one for mobile and tablet browsers. There were substantial differences in the way these two websites worked, and they shared no common platform. Each website had unique capabilities, but the desktop site provided a much larger superset of features compared to the mobile optimized site.

To move forward with enabling iDEAL and other new payment methods across multiple platforms, we quickly came to the conclusion that the best way forward was a unified approach to supporting multiple platforms using a single UI. This ultimately started us on our path towards responsive web design (RWD). Responsive design is a technique for delivering a consistent set of functionality across a wide range of screen sizes, from a single website. For our effort, we focused on the following goals:

• Enable access to all features and capabilities, regardless of device
• Deliver a consistent user experience that is optimized for device capabilities, including screen size and input method

Cross-functional alignment and prototyping

In order to successfully tackle a responsive design project, we first needed to answer the following question for our team: what is responsive design? In initial brainstorming meetings, we aligned around a common definition that emphasizes the use of CSS and JS to adapt a common user experience to varying screen sizes and input methods.

At Netflix, we like to move quickly and not let unnecessary process slow down our ability to innovate and move fast. Rapid design, rapid development. To kickstart this project, we assembled a core group of designers and user interface engineers, and held a weeklong workshop. The end goal was to produce a working prototype of a responsively designed signup flow.

This workshop approach allowed us to streamline the entire design and development process. Developers and designers working side by side to immediately tackle issues as they came up. In effect, we were pair programming. This allowed us to minimize the need for comps and wireframes, and develop straight in the browser. It provided the freedom to easily experiment with different design and engineering techniques, and identify common patterns that could be used across the entire signup flow. By the end of that week we had created a prototype of a fully functional, looks-good-on-whatever-device-you’re-using, Netflix signup experience.

A/B testing and rollout

With a functional responsive flow now built, we came to the final stretch. Being highly data-driven at Netflix, we wanted to measure that our changes were having a positive impact on how customers interact with the signup flow. So we cleaned up our prototype, turned it into a production quality experience and tested it globally against the current split stacks. We saw no impact to desktop signups, but we did a see an increase in conversion rates on phones and tablets due to the additional payment types we enabled for those devices. The results of our A/B test gave us the confidence to roll out this new signup flow as the default experience in all markets, and retire our separate phone/tablet stack.

App integration

However, we didn’t stop there. We had also supported signup from within our Android app prior to this project. Following our global rollout for browser-based signups, we quickly integrated our newly responsive signup experience into our Android app as an embedded web view. This enabled us to get much more leverage out of our responsive design investment. More about this unique approach in a future post.

Many devices, one platform

Whether on a 30” screen or a 4” one, our customers are now provided with an experience that works well and looks great across a wide range of devices. Development of the signup experience has been streamlined, increasing developer productivity. We now have a single platform that serves as foundation for all signup A/B testing and innovation and our customers are afforded the same options regardless of device.

Our journey towards responsive design has not ended, however. As device platforms evolve, and as user expectations change, our designers and engineers are constantly working towards enhancing cross-platform experiences for our customers.

What’s next?

In upcoming posts, we will further explore how we built our responsive signup flow. Some of the topics we will cover are: a deeper dive into the client and server-side techniques we used to integrate our browser experience into our Android app, a more in-depth look at the decisions we made in regards to mobile-first vs desktop-first approaches, and a review of the challenges in dealing with responsive images.

Can’t wait until the next post? Interested in learning more about responsive design? Come see Chris Saint-Amant, Netflix UI Engineering Manager, discuss the next generation of responsive design at SXSW Interactive in Austin, TX on March 8th.

Are you interested in working on cross-platform design and engineering challenges? We’re hiring front-end innovators to helps us realize our vision. Check out our jobs.

• Matt Bookman from Google shared how to leverage NetlixOSS Lipstick on top of Google Compute Engine.   Lipstick combines a graphical depiction of a Pig workflow with information about the job as it executes, giving developers insight that previously required a lot of sifting through logs (or a Pig expert) to piece together.
• Andrew Spyker from IBM shared how they’re leveraging many of NetflixOSS components on top of the SoftLayer infrastructure to build real-world applications - beyond the AcmeAir app that won last year’s Cloud Prize.
• Peter Sankauskas from Answers4AWS talked about the motivation behind his work on NetflixOSS components setup automation, and his work towards 0-click setup for many of the components.
• Daniel Chia from Coursera shared how they utilize NetflixOSS Priam and Aegithus to work with Cassandra.

• Zeno - in memory data serialization and distribution platform
• Suro - a distributed data pipeline which enables services to move, aggregate, route and store data
• STAASH - a language-agnostic as well as storage-agnostic web interface for storing data into persistent storage systems
• A preview of Dynomite - a thin Dynamo-based replication for cached data
• Aegithus - a bulk data pipeline out of Cassandra
• PigPen - Map-Reduce for Clojure
• S3mper - a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index.
• A preview of Inviso - a performance focused Big Data tool

• Declaratively build complex events out of simple events (ex. drag and drop)
• Coordinate and sequence multiple Ajax requests
• Reactively update the UI in response to data changes
• Eliminate memory leaks caused by neglecting to unsubscribe from events
• Gracefully propagate and handle asynchronous exceptions

varHuman=Class.extend({
    init:function(height, weight){
this.height = height;
this.weight = weight;
}
});
varMutant=Human.extend({
    init:function(height, weight, abilities){
this._super(height, weight);
this.abilities = abilities;
}
});

• When extending an object, it loops over the prototype to find functions that use _super

• To find _super it decompiles a function into a string and tests that string against a regular expression.
• Any function using _super is then wrapped in a closure to achieve the super-class chaining.

The _super implementation basically uses the wrapping closure as a complex pointer with references to the overriding function and the inherited function: 

If we could find a way to remove the middle man, we’d be able to have the best of both worlds.

for(name in source){
    value = source[name];
    currentValue = receiver[name];
if(typeof value ==='function'&&typeof currentValue ==='function'&&
        value !== currentValue){
        value.base = currentValue;
}
}

varHuman=Class.extend({
    init:function(height, weight){
this.height = height;
this.weight = weight;
}
});
varMutant=Human.extend({
    init:functioninit(height, weight, abilities){
init.base.call(this, height, weight);
this.abilities = abilities;
}
});

var theWolverine =newMutant('5ft 3in',300,[
'adamantium skeleton',
'heals quickly',
'claws'
]);

by: Nicholas Eddy

Our 48 million members are accustomed to seeing a screen like this, whether on their TV or one of the 1000+ other Netflix devices they enjoy watching on. But the simple act of pressing play calls into action a deep and complex system that handles the DRM licenses, contract evaluations, CDN selection, and more. This system is known internally as the Playback Service and is responsible for making your Netflix streaming experience seem effortless.

The original Playback Service was built before Netflix was synonymous with streaming.  As our product matured, the existing architecture became more difficult to support and started showing signs of stress.  For example, we debuted HTML5 support last summer on IE11 in a major step towards standards-based playback on web platforms.  This required adoption of a new device security model that works with the emerging HTML5 Web Cryptography API.  It would have been very challenging to integrate this new model into our original architecture due to poor separation of business and security logic.  This and other shortcomings pushed us to re-imagine our design, leading to a radically different solution with the following as our key design requirements:

• Operation at massive scale
• High velocity innovation
• Reusability of components

High-level Architecture

---

The new Playback Service uses self-assembling components to handle the enormous volume of traffic Netflix gets each day.  The following is an overview of that architecture; with special focus given to how requests are delegated to these components which are dynamically assembled and executed within an internal rule engine.

We will examine the building blocks of this architecture and show the benefits of using small, reusable components that are automatically wired together to create an emergent system.  This post will focus on the smallest units of the new architecture and the ideas behind self-assembly.  It will be followed up by others that go deeper into how we implement these concepts in the new architecture and address some challenges inherent to this approach.

Bottom-up

---

We started from the bottom up; defining the building blocks of the new architecture in a way that promotes loose coupling and clear separation of concerns.  These building blocks are called Processors: entities that take zero or more inputs and generate no more than one output.  These are the smallest computational unit of the architecture and behave like commands ala The Gang of Four.  Below is a diagram that shows a processor that takes A, B, C and generates D.

This metaphor generalizes well given that many complex tasks can be subdivided into discrete, function-like steps.  It also matches the way most engineers already think about problem decomposition.  This definition enables processors to be as specialized as necessary, promoting low interconnectedness with other parts of the system.  These qualities make the system easier to reason about, enhance, and test.

The Playback Service--like other complex systems--can be modelled as a black box that takes inputs and generates outputs.  The conversion of some input A to an output E can be defined as a function f(A) = E and modelled as a single processor.

Of course, using a single processor only makes sense for very simple systems.  More complex services  would be decomposed into finer-grained processors as illustrated below.

Here you can see that the computation of Eis handled as several processor invocations.  This flow resembles a series of function calls in a Java program, but there are some fundamental differences.  The difficulty with normal functions is someone has to invoke them and decide how they are wired together.  Essentially, the decomposition of f(A) = E above is usually something the entire team needs to understand and maintain.  This places a cap on system evolution since scaling the system means scaling each engineer.  It also increases the cost of scaling the team since minimum ramp-up time is directly proportional to the system complexity.

But what if you could have functions that self-assemble?  What if processors could simply advertise their inputs/outputs and the wiring between them were an emergent property of that particular collection of processors?

Self-assembling Components

---

The hypothesis is that complex systems can be built efficiently if they are reduced to small, local problems that are solved in relative isolation with processors.  These small blocks are then automatically assembled to reveal a fully formed system.  Such a system would no longer require engineers to understand their entire scope before making significant contributions.  These systems would be free to scale without taxing their engineering teams proportionally.  Likewise, their teams could grow without investing in lots of onboarding time for each new member.

We can use the decomposition we did above for f(A) = E to illustrate how a self-assembly would work.  Here is a simplified version of the diagram we saw earlier.

This system solves for A => E using the processors shown.  However, this could be a more sophisticated system containing other processors that do not participate in the computation of E given A.  Consider the following, where the system’s complete set of processors is included in the diagram.

The other processors are inactive for this computation, but various combinations would become active under different inputs.  Take a case where the inputs were J, and W and processors were in place to handle these inputs such that the computation J,W => Y were possible.

The inputs J and W would trigger a different set of processors than before; leaving those that computed A => E dormant.

The processors triggered for some inputs is an emergent property of the complete set of processors within the system.  An assembler mechanism exists to determine when each processor can participate in the computation.  It makes this decision at runtime, allowing for a fully dynamic wiring for each request.  As a result, processors can be organized in any way and do not need to be aware of each other.  This makes their functionality easier to add, remove, and update than conventional mechanisms like switch statements or inheritance; which are statically determined and more rigidly structured.

Extending traditional systems often means ramping up on a lot of code to understand where the relevant inflection points are for a change or feature.  Self-assembly relaxes the urgency for this deeper context and shifts the focus towards getting the right interaction designs for each component.  It also enables more thorough testing since processors are naturally isolated from each other and simpler to unit test.  They can also be assembled and run as a group with mocked dependencies to facilitate thorough end-to-end validation.

Self-assembly frees engineers to focus on solving local problems and adding value without having to wrestle with the entire end-to-end context.  State validation is a good example of an enhancement that requires only local context with this architecture.  The computation of J,W => Y above can be enhanced to include additional validation of V whenever it is generated.  This could be achieved by adding a new processor that operates on V as an input: illustrated below.

The new processor V => V would take a value and raise an error if that value is invalid for some reason.  This validation would be triggered whenever V is present in the system, whether or not J,W => Y is being computed.  This is by design; meaning each processor is reused whenever its services are needed.

This validator pattern emerges often in the new Playback Service.  For example, we use it to detect whether data sent by clients has been tampered with mid-flight.  This is done using HMAC calculations to verify the data matches a client provided hash value.  As with other processors, the integrity protection service provided this way is available for use during any request.

Challenges of Self-assembly

---

The use of self-assembling components offers clear advantages over hand wiring.  It enables fluid architectures that can change dynamically at runtime and simplifies feature isolation so components can evolve rapidly with minimal impact to the overall system.  Moreover, it decouples team size from system complexity so the two can scale independently.

Despite these benefits, building a working solution that enables self-assembly is non-trivial.  Such a system has to decide which operations are executed when, and in what order.  It has to manage the computation pipeline without adding too much overhead or complexity; all while scaling up with the set of processors.  It also needs to be relatively unobtrusive so developers can remain focused on building the service.  These were some of the challenges my team had to overcome when building the new Playback architecture atop the concepts of self-assembly.

Upcoming...

---

Subsequent blog posts will take us deeper into the workings of the new Playback Service architecture and provide more details about how we solved the challenges above and other issues intrinsic to self-assembly.  We will also be discussing how this architecture is designed to enable fully dynamic end-points (where the set of rules/processors can change for each request) as well as dynamic services where the set of end-points can change for a running server.

The new Playback Service architecture based on self-assembling components provides a flexible programming model that is easy to develop and test.  It greatly improves our ability to innovate as we continue to enhance the viewing experience for our members.

We are always looking for talented engineers to join us.  So reach out if you are excited about this kind of engineering endeavor and would like to learn more about this and other things we are working on.

• Understanding the impact of QoE on user behavior
• Creating a personalized streaming experience for each member
• Determining what movies and shows to cache on the edge servers based on member viewing behavior
• Improving the technical quality of the content in our catalog using viewing data and member feedback

At Netflix, responsibility for delivering the streaming service is distributed and the environment is constantly changing. Code is deployed thousands of times a day, and cloud configuration parameters are modified just as frequently. To understand and manage the risk associated with this velocity, the security team needs to understand how things are changing and how these changes impact our security posture.

Overview of Security Monkey

Viewing IAM users in Security Monkey - highlighted users have active access keys.

Security Monkey's filter interface allows you to quickly find the configurations and items you're looking for.

Architecture

• Watcher - The component that monitors a given AWS account and technology (e.g. S3, IAM, EC2). The Watcher detects and records changes to configurations. So, if a new IAM user is created or if an S3 bucket policy changes, the Watcher will detect this and store the change in Security Monkey’s database.
• Notifier - The component that lets a user or group of users know when a particular item has changed. This component also provides notification based on the triggering of audit rules.
• Auditor - Component that executes a set of business rules against an AWS configuration to determine the level of risk associated with the configuration. For example, a rule may look for a security group with a rule allowing ingress from 0.0.0.0/0 (meaning the security group is open to the Internet). Or, a rule may look for an S3 policy that allows access from an unknown AWS account (meaning you may be unintentionally sharing the data stored in your S3 bucket). Security Monkey has a number of built-in rules included, and users are free to add their own rules.

General Features and Operations

• Initial Configuration
• Setting up one or more Security Monkey users to use/administer the application itself.
• Setting up one or more AWS accounts for Security Monkey to monitor.
• Configuring user-specific notification preferences (to determine whether or not a given user should be notified for configuration changes and audit reports).

• Typical Use Cases
• Checking historical details for a given configuration item (e.g. the different states a security group has had over time).
• Viewing reports to check what audit issues exist (e.g. all S3 policies that reference unknown accounts or all IAM users that have active access keys).
• Justifying audit issues (providing background or context on why a particular issues exists and is acceptable though it may violate an audit rule).

Note on AWS CloudTrail and AWS Trusted Advisor

• CloudTrail provides verbose data on API calls, but has no sense of state in terms of how a particular configuration item (e.g. security group) has changed over time. Security Monkey provides exactly this capability.
• Trusted Advisor has some excellent checks, but it is a paid service and provides no means for the user to add custom security checks. For example, Netflix has a custom check to identify whether a given IAM user matches a Netflix employee user account, something that is impossible to do via Trusted Advisor. Trusted Advisor is also a per-account service, whereas Security Monkey scales to support and monitor an arbitrary number of AWS accounts from a single Security Monkey installation.

Open Items and Future Plans

• Integration with CloudTrail for change detail (including originating IP, instance, IAM account).
• Ability to compare different configuration items across regions or accounts.
• CSRF protections for form POSTs.
• Content Security Policy headers (currently awaiting a Dart issue to be addressed).
• Additional AWS technology and configuration tracking.
• Test integration with moto.
• SSL certificate expiry monitoring.
• Simpler installation script and documentation.
• Roles/authorization capabilities for admin vs. user roles.
• More refined AWS permissions for Security Monkey operations (the current policy in the install docs is a broader read-only role).
• Integration with edda, our general purpose AWS change tracker. On a related note, our friends at Prezi have open sourced reddalert, a security change detector that is itself integrated with edda.

Conclusion

We wanted to create a space for exchanging information and learning among professionals. That space would serve as a forum, or an agora, for a community of people sharing the same interests in the engineering aspects of billing & payment systems.

• Mathieu Chauvin - Engineering Manager for Payments @ Netflix
• Taylor Wicksell - Sr. Software Engineer for Billing @ Netflix
• Jean-Denis Greze - Engineer @ Dropbox
• Alec Holmes - Software Engineer @ Square
• Emmanuel Cron - Software Engineer III, Google Wallet @ Google
• Paul Huang - Engineering Manager @ Survey Monkey
• Anthony Zacharakis - Lead Engineer @ Lumos Labs
• Shengyong Li / Feifeng Yang - Dir. Engineering Commerce / Tech Lead Payment @ Electronic Arts