On of the fundamental problems in reinforcement learning is handling exploration exploitation tradeoff. Distributional reinforcement learning estimates and maintains a distribution over returns and has shown state of the art performance in may environments. We try to make use of the extra information available in distributional RL for efficient exploration. We leverage on the uncertainty of the unexplored states to direct our agent for efficient exploration. we experimented with environments from OpenAI gym and exploratory chain task which is considered as an difficult exploratory problem