pykafka.partitioners¶
Author: Keith Bourgoin, Emmett Butler
-
pykafka.partitioners.
random_partitioner
(partitions, key)¶ Returns a random partition out of all of the available partitions.
-
class
pykafka.partitioners.
BasePartitioner
¶ Bases:
object
Base class for custom class-based partitioners.
A partitioner is used by the
pykafka.producer.Producer
to decide which partition to which to produce messages.-
__weakref__
¶ list of weak references to the object (if defined)
-
-
class
pykafka.partitioners.
HashingPartitioner
(hash_func=<built-in function hash>)¶ Bases:
pykafka.partitioners.BasePartitioner
Returns a (relatively) consistent partition out of all available partitions based on the key.
Messages that are published with the same keys are not guaranteed to end up on the same broker if the number of brokers changes (due to the addition or removal of a broker, planned or unplanned) or if the number of topics per partition changes. This is also unreliable when not all brokers are aware of a topic, since the number of available partitions will be in flux until all brokers have accepted a write to that topic and have declared how many partitions that they are actually serving.
-
__call__
(partitions, key)¶ Parameters: - partitions (sequence of
pykafka.base.BasePartition
) – The partitions from which to choose - key (Any hashable type if using the default
hash()
implementation, any valid value for your custom hash function) – Key used for routing
Returns: A partition
Return type: pykafka.base.BasePartition
- partitions (sequence of
-
__init__
(hash_func=<built-in function hash>)¶ Parameters: hash_func (function) – hash function (defaults to hash()
), should return an int. If hash randomization (Python 2.7) is enabled, a custom hashing function should be defined that is consistent between interpreter restarts.
-