A lack of obvious, short-term "fruit" often leads people to question the faithfulness of the pastor. This may be appropriate, and indeed, accountability to remain faithful is a hallmark of a good pastor.
But too often a lack of short-term "fruitfulness" weakens a minister's resolve to endure, even though a Biblical call for endurance is exactly what is needed in so many cases.
What about the fruitfulness of prayer? Or the fruitfulness of obedience? Jeremiah, Hosea, and others were faithful in their calling, but none of them saw "fruit" in the way we usually think of it. They called people to repentance and no one repented. Yet God seems to make much of their fruitfulness of obedience, their fruitfulness of endurance, their fruitfulness of faithfulness, their fruitfulness of doing what is right even if the stars fall.
Faithfulness Without Seeing Fruitfulness
Sometimes pastors can be critized for the lack of fruit in their ministries or the church where they pastor. However, that is not an indication of a lack of faithfulness. Here is an excerpt from an excellent post on this issue.