Large language models (LLMs), such as ChatGPT, have substantially helped in understanding human inquiries and generating textual content with human-level fluency. However, directly using LLMs in healthcare applications faces several problems. LLMs are prone to produce hallucinations, or fluent content that appears reasonable and genuine but that is factually incorrect. Ideally, the source of the generated content should be easily traced for clinicians to evaluate. We propose a knowledge-grounded collaborative large language model, DrugGPT, to make accurate, evidence-based and faithful recommendations that can be used for clinical decisions. DrugGPT incorporates diverse clinical-standard knowledge bases and introduces a collaborative mechanism that adaptively analyses inquiries, captures relevant knowledge sources and aligns these inquiries and knowledge sources when dealing with different drugs. We evaluate the proposed DrugGPT on drug recommendation, dosage recommendation, identification of adverse reactions, identification of potential drug-drug interactions and answering general pharmacology questions. DrugGPT outperforms a wide range of existing LLMs and achieves state-of-the-art performance across all metrics with fewer parameters than generic LLMs.