News
Copilot introduced in your organization? Better check user activity
Many organizations have jumped on the Gen-AI bandwagon in a big way, immediately hiring AI leaders and deploying AI assistants to many – if not all – of their employees. For us here at Visual Studio Magazinethat is, developers who were likely granted access to GitHub Copilot, the original “AI pair programmer” that ushered in Microsoft’s “AI era” years ago.
But now that advanced AI has reached the “trough of disillusionment” stage of the hype cycle, it may be time to review user activity for GitHub Copilot in your organization, both among developers and other employees who use one of Microsoft’s seemingly countless Copilots across a wide range of products and services.
Reviewing this data can help a company optimize its investment in Copilot and ensure it effectively supports its development processes. Potential benefits include:
- Resource allocation: Companies can ensure efficient use of copilot licenses by monitoring active usage and reallocating seats when needed to maximize productivity.
- Cost management: By tracking usage, companies can manage subscription costs and potentially downgrade or upgrade based on actual needs.
- Security and Compliance: The monitoring activity ensures that Copilot usage complies with company policies and security standards.
Fortunately, current documentation from Microsoft and Microsoft’s own GitHub provides guidance on how to review user activity data for GitHub Copilot in an organization.
For example, a few weeks ago Microsoft released “Microsoft 365 Reporting in the Admin Center – Copilot for Microsoft 365 Usage.”
It states: “The Microsoft 365 usage dashboard shows you the activity overview across the Microsoft 365 productivity apps in your organization. It allows you to drill into individual product-level reports to get a more detailed look at the activity within each app.”
Managers can view a summary of user adoption, retention, and usage of Copilot for Microsoft 365, as well as the activity of each Copilot user in the organization. Reports are typically available within 72 hours of the end of a day’s activity.
Speaking of dashboards: Earlier this year, Microsoft announced: “Microsoft Copilot Dashboard now generally available.”
The announcement states: “Today we are excited to announce the general availability of the Microsoft Copilot Dashboard with Viva Insights, a new and improved way to understand the impact of your investment in Copilot for Microsoft 365. Originally announced at Microsoft Ignite 2023, the Copilot Dashboard provides privacy-protected insights into every stage of the transformation journey and provides organizations with the ability to understand the impact of Copilot on common workplace activities such as meetings, email, chat, documents, and search.”
Just last week, Microsoft published a guide entitled “Connecting to the Microsoft Copilot dashboard for Microsoft 365 customers.”
It states: “The Microsoft Copilot Dashboard in Viva Insights helps organizations maximize the value of Copilot for Microsoft 365. It provides actionable insights to help your organization prepare for AI deployment, drive adoption based on how AI is changing workplace behavior, and measure the impact of Copilot.”
Other guidance from Microsoft includes “Analyze Copilot performance and usage in Copilot Studio,” which provides overview charts and data on engagement over time, session outcomes over time, resolution rate drivers, and more.
For example, it states: “The Summary tab of the Analytics page gives you a comprehensive overview of your copilot’s performance. It uses artificial intelligence (AI) to show you which topics have the greatest impact on escalation rate, abandonment rate, and resolution rate.”
“Inappropriate or risky interactions”
In addition to reviewing usage data for cost control, efficient resource allocation, etc., Microsoft Purview’s guide also includes “Configuring a communications compliance policy to detect Copilot for Microsoft 365 interactions.”
It states: “Communications compliance enables you to analyze interactions (prompts and responses) entered in Copilot for Microsoft 365 to identify inappropriate or risky interactions or the sharing of sensitive information.”
It works with a variety of copilot apps:
- Excel Copilot
- Forms Copilot
- Loop Copilot
- Microsoft 365 Chat on Bing
- Microsoft 365 Chat in Teams
- OneNote Copilot
- Outlook Copilot
- Planner Copilot
- PowerPoint Copilot
- StreamCopilot
- Teams (Chats/Channels/Meetings) Copilot
- Word Copilot
- Whiteboard Copilot
Interactions covered include:
GitHub speaks out
GitHub has since released “Reviewing User Activity Data for Copilot in Your Organization” to help “review GitHub Copilot usage in your organization to make informed decisions about seat allocation.”
This can be helpful in cost management.
GitHub has also released “View Copilot license usage across your organization” to “show how many users across all organizations in your company have access to GitHub Copilot.”
It states: “If you have a subscription to GitHub Copilot Business or GitHub Copilot Enterprise, you can view usage information for GitHub Copilot in your company, broken down by organization, or in your organization, broken down by seat assignment status. At the company level, this information includes the number of seats assigned in each organization and the total spend associated with each organization for the current billing cycle. At the organization level, this information includes the total number of seats, seats carried over from the previous billing cycle, new seats added during the current cycle, and seats to be removed at the end of the current cycle.”
Microsoft Copilot speaks out
We asked Microsoft Copilot why an organization would want to track Copilot usage activity. The answer was:
Organizations track GitHub Copilot usage activity for several important reasons:
- Resource optimization
By monitoring Copilot usage, companies can ensure the tool is being used effectively. This helps reallocate resources to those who will benefit most and ensures that the investment in Copilot is justified and maximized.- Cost management
Usage tracking can identify inactive or underused licenses, allowing organizations to manage their subscriptions more efficiently and potentially reduce costs by reassigning or canceling unnecessary licenses.- Productivity insights
Understanding how Copilot impacts productivity can provide valuable insights. Companies can assess whether the tool helps developers code more efficiently and identify areas where additional training or support may be needed to increase effectiveness.- Security and Compliance
Monitoring user activity can help identify unusual or unauthorized usage patterns and ensure that the company’s security and compliance standards are met. This is especially important for protecting sensitive code and data.- User engagement and support
By tracking which team members are actively using Copilot, organizations can identify those who may need additional support or training. This ensures that all team members can take full advantage of the tool’s features, resulting in better overall performance.- Informed decision making
By accessing detailed usage data, companies can make data-driven decisions regarding tool adoption and usage. This can lead to better strategic planning and more effective implementation of development tools.
About the author
David Ramel is an editor and writer for Converge360.
